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Preface 


The past few decades have witnessed tremendous development in the manu- 
facture of computers and Software, and scientific computing has become an 
important tool for finding Solutions to scientific problems that come from var- 
ious branches of Science and engineering. Nowadays, scientific computing has 
become one of the most important means of research and learning in the fields 
of Science and engineering, which are indispensable to any researcher, teacher, 
or student in the fields of Science and engineering. 

One of the most important branches of scientific computing is a numer- 
ical analysis which deals with the issues of finding approximate numerical 
Solutions to such problems and analyzing errors related to such approximate 
methods. Both the MATLAB® and Python programming languages provide 
many libraries that can be used to find Solutions of scientific problems visual- 
izing them. The ease of use of these two languages became the most languages 
that most scientists who use computers to solve scientific problems care about. 

The idea of this book came after I taught courses of scientific computing for 
physics students, introductory and advanced courses in mathematical Software 
and mathematical computer applications in many Universities in Africa and 
the gulf area. I also conducted some workshops for mathematics and Science 
students who are interested in computational mathematics in some Sudanese 
Universities. In these courses and workshops, MATLAB and Python were used 
for the implementation of the numerical approximation algorithms. Hence, the 
purpose of introducing this book is to provide the student with a practical 
guide to solve mathematical problems using MATLAB and Python Software 
without the need for third-party assistance. Since numerical analysis is con- 
cerned with the problems of approximation and analysis of errors of numerical 
methods associated with approximation methods, this book is more concerned 
with how these two aspects are applied in practice by Software, where illustra- 
tions and tables are used to clarify approximate Solutions, errors and speed of 
convergence, and its relations to some of the numerical method parameters, 
such as step size and tolerance. MATLAB and Python are the most popular 
programming languages for mathematicians, scientists, and engineers. Both 
the two programming languages possess various libraries for numerical and 
symbolic computations and data representation and visualization. Proficiency 
with the computer programs contained in this book requires that the student 
have prior knowledge of the basies of the programming languages MATLAB 
and Python, such as branching, Loops, symbolic packages, and the graphical 


xiii 
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libraries. The MATLAB version used for this book is 2017b and the Python 
version is 3.7.4. 

The book consists of 11 chapters divided into three parts: the first part is 
concerned with discussing numerical Solutions for linear and nonlinear Systems 
and numerical difficulties facing these types of problems with how to overcome 
these numerical difficulties. The second part deals with methods of completing 
functions, differential and numerical integration, and Solutions of differential 
equations. The last part of the book discusses methods to solve linear and 
nonlinear programming and optimal control problems. It also contains some 
specialized Software in Python language to solve some problems numerically. 
These Software packages must be downloaded from a third party, such as 
Gekko which is used for the Solutions of differential equations and linear and 
nonlinear programming in addition to the optimal control problems. Also, the 
Pulp package is used to solve linear programming problems and finally Pyomo 
a package is used for solving linear and nonlinear programming problems. How 
to install and run such a package is also presented in the book. 

What distinguishes this book from many other numerical analysis books 
is that it contains some topics that are not usually found in other books, such 
as nonstandard finite difference methods for solving differential equations and 
Solutions of optimal control problems. In addition, the book discusses imple- 
mentations of methods with high convergence rates, such as Gauss integration 
methods discussed in the numerical differentiation and integration, exact finite 
difference schemes for solving differential equations discussed in the nonstan¬ 
dard finite differences Ghapter. It also uses efficient python-based Software for 
solving some kinds of mathematical problems numerically. 

The parts of the book are separate from each other so that the student 
can study any part of it without having to read the previous parts of that 
part. The exception to this is the optimal control chapter in the third part, 
which requires studying numerical methods to solve the differential equations 
discussed in the second part. 

After reading this book and implementing the programs contained on it, 
a student will be able to deal with and solve many kinds of mathematical 
problems such as differential equations, static, and dynamical optimization 
problems and apply the methods to real-life problems. 
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Solving Linear Systems Using Direct Methods 


Abstract 

Linear Systems of equations have many applications in mathematics and Sci¬ 
ence. Many of the numerical methods used for solving mathematics problems 
such as differential or integral equations, polynomial approximations of tran- 
scendental functions and solving Systems of nonlinear equations arrive at a 
stage of solving linear Systems of equations. Hence, solving a linear system of 
equations is a fundamental problem in numerical computing. 

This chapter discusses the direct methods for solving linear Systems of 
equations, using Gauss and Gauss-Jordan elimination techniques and the 
matrix factorization approach. MATLAB® and Python implementations of 
such algorithms are provided. 


1.1 Testing the Existence of the Solutiori 

A linear system consisting of m equations in n unknowns, can be written in 
the matrix form: 

Ax = b (1-1) 

where. 


A = 

foii 012 

021 021 

(X\-Yi ^ 
^2n 


fxi\ 

X2 

and b = 

/h\ 

b2 


^OttiI Clrn2 

^mn ) 


\^n j 




Here, the coefhcients aij of matrix A G Rrrexn assumed to be real, 
X gMA is the vector of unknowns and b G is a known vector. Depending 
on the relationship between m and n three kinds of linear Systems are defined 
[30, 53]: 

1. overdetermined linear systems: there are more equations than 
unknown (rn> n). 


3 
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2. determined linear systems: equal numbers of equations and unknowns 
(m = n). 

3. underdetermined linear systems: there are more unknowns than equa¬ 
tions (to < n). 

Let A = [A I b] be the augmented matrix of the linear system Ax = b. 
Then, the existence of a solution for the given linear system is subject to one 
of the two following cases: 

1. rank{A) = rank{A)\ in this case, there is at least one solution, and we 
have two possibilities: 

(a) rank{A) = rank{A) = n\ in this case there is a unique solution. 

(b) rank{A) = rank(A) < n: in this case there is infinite number of Solu¬ 
tions. 

2. rank{A) > rank(A): in this case there is no solution and we can look for 
a least squares solution. 

If the linear system Ax = b has a solution, it is called a consistent linear 
System, otherwise, it is an inconsistent linear system [30]. 

In MATLAB, the command rank can be used to test the rank of a given 
matrix A. 

» A = [1 2 3; 4 5 6; 7 8 9] 

A = 

1 2 3 

4 5 6 

7 8 9 

» b = [1; 1; 1] 
b = 

1 

1 

1 

» rl = rank(A) 
rl = 

2 

» r2 = rank([A b]) 
r2 = 

2 

In python, the function matrixjrank (located in numpy. linalg) is used to 
compute the rank of matrix A and the augmented system [Ab\. 

In [1]: import numpy as np 

In [2]: A = np. arrayC [ [1, 2, 3], [4, 5, 6], [7, 8, 9]]) 

In [3]: b = np.array([l, 1, 1]) 
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In [4]; rl, r2 = np.linalg.matrix_rank(A), np.linalg.matrix_rank 
(np.c_[A, b]) 

In [5] : rl 
Out [5] : 2 
In [6] : r2 
Out [6] ; 2 

In the special case when m = n {A is a squared matrix) and there is a unique 
solution (rank{A) = rank{A) = n), this unique solution is given by: 

X = A~^b. 

Hence, finding the solution of the linear system requires the inversion of 
matrix A. 


1.2 Methods for Solving Linear Systems 

This section considers three special types of linear Systems which are linear 
Systems with diagonal, upper triangular and lower triangular matrices. 

1.2.1 Special Linear Systems 

We consider the linear system: 


Ax = b, 


where A G x and b G R”". We consider two cases. 


1. Gl is a diagonal matrix: 

In this case, matrix A is of the form: 



/ Oll 

0 

0 

... 0 \ 


0 

022 

0 

... 0 

A = 

0 

0 

033 

... 0 


U 

0 

0 

^nn / 


which leads to the linear system: 


/ «11 
0 
0 


Vo 


0 

022 

0 

0 


0 

0 

033 


0 \ 

0 

0 


/xi\ 

X2 

X3 


Onn / \Xn / 


/6i\ 

b2 

H 

\bnj 


( 1 . 2 ) 
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The solution of the linear system (1.2) is given by: 

Xi = — 

The MATLAB code to compute this solution is given by: 


1 function x = SolveDiagonalLinearSystem(A, b) 

2 % This function solves the linear system Ax = b, where ... 

A is a diagonal matrix 

3 % b is a known vector and n is the dimension of the ... 

problem . 

4 n = length(b) ; 

5 X = zeros(n, 1) ; 

6 for j = 1: n 

7 x{j) = b(j) /A(j, j) ; 

8 end 


We can apply this function to solve the diagonal system: 



by using the following MATLAB commands: 

» A = diag([2, -1, 3]) 

A = 

2 0 0 

0-10 

0 0 3 

» b = [4; 1; 3] 
b = 

4 

1 

3 

>> X = SolveDiagonalLinearSystemCA, b) 

X = 

2 

-1 

1 

The python code of the function SolveDiagonalLinearSystem is as fol- 
lows. 


1 import numpy as np 

2 def SolveDiagonalLinearSystem(A, b): 

3 n = len (b) 
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4 X = np.zeros((n, 1), 'float') 

5 for i in range(n) : 

6 X [i] = b [i] /A[i, i] 

7 return x 


In [7]: A = np.diag([2, -1, 3]) 

In [8]: b = np.array([4, -1, 3]) 

In [9]; X = SoIveDiagonaILinearSystem(A, b) 
In [10]; print(’x = \n’, x) 

X = 

[[ 2 .] 

[ 1 .] 

[ 1.11 


2. is an upper triangular matrix: 

In this case, matrix A is of the form: 


A = 


/ Oll 
0 
0 


012 

022 

0 


013 

023 

033 


Oln^ 

02n 

^3n 


yO 0 0 ... ann/ 

Therefore, we have the linear system: 


/ Oll 

012 

013 • 

Gln 


< Xi\ 


/h\ 

0 

022 

023 • 

^2n 


X2 


b2 

0 

0 

033 • 

^3n 


X3 

= 

bs 

V 0 

0 

0 . 

^nn j 


\Xn/ 


\bnj 


(1.3) 


In this case we use the back substitution method for finding the solution 
of System 1.3. The MATLAB function SoIveUpperSystem.m solves the 
linear system 1.3 using the back-substitution method. 


1 function x = SolveUpperLinearSystem(A, b) 

2 % This function uses the backward substitution method ... 

for solving 

3 % the linear system Ax = b, where A is an upper ... 

triangular matrix 

4 % b is a known vector and n is the dimension of the ... 

problem . 

5 n = length(b) ; 

6 X = zeros(n, 1) ; 

7 X(n) = b(n)/A{n, n) ; 

8 for j = n-1: -1 : 1 

9 X {j) = b ( j ) ; 
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10 for k = j + 1 : n 

11 x(j) = x{j) - A(j, k)*x{k) ; 

12 end 

13 X { j ) = X ( j ) /A ( j , j ) ; 

14 end 


The python code for the SolveUpperSystem, is as follows. 


1 import numpy as np 

2 def SolveUpperLinearSystem(A, b): 

3 n = len (b) 

4 X = np.zeros((n, 1), 'float') 

5 x[n-l] = b[n-1]/A[n-1, n-1] 

6 for i in range(n-2, - 1 , -1): 

7 x[i] = b[i] 

8 for j in range(i + l, n) : 

9 x[i]-=A[i,j]*x[j] 

10 x[i]/=A[i,i] 

11 return x 


3. A is a lower triangular system: 

In this case, matrix A is of the form: 



/ Oll 

0 

0 . 

■ \ 


021 

«22 

0 . 

0 

A = 

«31 

«32 

«33 ■ 

0 


^«nl 

^n2 

^n3 

^nn j 

Therefore, we have the linear 

System: 




^«11 

0 

0 . 

■ ^ ^ 


f Xl\ 


(hA 

«21 

«22 

0 . 

0 


X2 


b2 

«31 

«32 

«33 • 

0 


X3 

= 

bs 


^ n 2 

^ n 3 

^nn j 


\Xn j 


\bnj 


(1.4) 


The forward substitution method is used to find the solution of system 1.4. 
The MATLAB function SolveLowerSystem.m solves the linear system 1.4 
using the forward-substitution method. 


1 function x = SolveLowerLinearSystem(A, b) 

2 % This function uses the forward substitution method ... 

for solving 

3 % the linear system Ax = b, where A is an lower ... 

triangular matrix 

4 % b is a known vector and n is the dimension of the ... 

problem . 
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5 n = length(b) ; 

6 X = zeros(n, 1) ; 

7 x{l) = b(l) /A{1, 1) ; 

8 for j = 2 : n 

9 X {j) = b ( j ) ; 

10 for k = 1 : j-1 

11 x(j) = x{j) - A(j, k)*x{k) ; 

12 end 

13 X { j ) = X ( j ) /A ( j , j ) ; 

14 end 


The python code of the function SolveLowerSystem is as follows. 


1 

def SolveLowerLinearSystem(A, b): 

2 

import numpy as np 

3 

n = 

len (b) 

4 

X = 

np.zeros((n, 1), 'float') 

5 

x[0 

= b[0] /A[0, 0] 

6 

for 

i in range (1, n): 

7 


x[i] = b[i] 

8 


for j in range (i): 

9 


x[i] -= A[i, j] *x[ j] 

10 


x[i] /= A[i, i] 

11 

return x 


1.2.2 Gauss and Gauss-Jordan Elimination 

Gauss and Gauss-Jordan elimination methods are related to each other. If 
given a matrix A G then both Gauss and Gauss-Jordan apply ele- 

mentary row operations through consequent steps over matrix A. The Gauss 
method stops after obtaining the row echelon form of matrix A (If A is non- 
singular, then its row echelon form is an upper triangular matrix), whereas 
Gauss-Jordan continuous until reaching the reduced row echelon form (If 
A is nonsingular, then its reduced row echelon form is the identity matrix). 

To illustrate the differences between the row echelon and the reduced row 
echelon forms, the two forms are computed for the matrix: 



Starting by finding the row echelon form for the given matrix. 



4 

-1 


/4 

-1 


0 

15 

- 5 ) 


15 

-5 

0 

-5 

15/ 

lo 

0 

40 
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The upper triangular matrix 

/4 -1 -1\ 

0 15 -5 

\0 0 40 / 

is the row echelon form of matrix A. 

Gauss-Jordan elimination continues above the pivot elements, to obtain 
the reduced row echelon form. 


4 -1 
0 15 

0 0 

R2<-R2/15 



, /4 -1 

fo 15 

\0 0 


-1 

1 

0 





—i?2-l-5 i^3 


-1 

15 

0 


/4 0 
0 1 
\0 0 


oj 


0 

1 

0 


1.2.3 Solving the System with the rref Function 

The Gauss and Gauss-Jordan methods are two familiar approaches for solv¬ 
ing linear Systems. Both begin from the augmented matrix, obtain the row 
echelon form or the reduced row echelon form, respectively. Then, the Gauss 
method uses the back-substitution technique to obtain the solution of the lin¬ 
ear System, whereas in Gauss-Jordan method the solution is located in the 
last column. 

The MATLAB code below, reads a matrix A and a vector b from the user, 
then it applies the Gauss-Seidel elimination through applying the rref to the 
augmented System [A b] 


1 ciear ; clc ; 

2 A = input('Enter the matrix A: ') ; % Reading matrix A from ... 

the user 

3 b = input('Enter the vector b: ') ; % Reading vector b from ... 


[m. 

the user 

n] = size (A) ; 

% 

m and n . 

are 

the matrix 

rl 

dimensions 
= rank (A) ; 

% 

the rank 

of 

matrix A is 

r2 

assigned to rl 
= rank ([Ab]) ; 

% 

the rank 

of 

the ... 


augmented system [A b] is assigned to r2 

7 if rl ^ r2 % testing whether rank (A) ... 

not equal rank ( [A b] ) 

8 disp ( [ ' Rank (A) = ' num2str(rl) ' ' num2str(r2) ' = ... 

Rank([Ab]).']) ; 

9 fprintf (' There is no solution.\n' ) ; % No solution in this ... 

case 

10 end 

11 if rl == r2 % testing whether rank(A) = ... 

rank([A b]) 
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12 if rl == n % if yes, testing whether the ... 

rank equals n 

13 R = rref ([Ab]); % the reduced row echelon form ... 

of [A b] 

14 X = R(:, end) ; % the solution is at the last ... 

column of the reduced 

15 % row echelon form 

16 disp {[ 'Rank(A) = Rank([A b]) = ' num2str(rl) ' = 

#Col(A).'] ) ; 

17 disp( 'There is a unique solution, given by: ') ; ... 

disp (x) ; 

18 %displaying the solution of the linear system 

19 else % rank(A) = rank ( [A b]) < n 

20 disp {[' Rank(A) = Rank([A b]) = ' num2str(rl) ' < ' ... 

num2str (n) ' = #Col(A).’ ]) ; 

21 fprintf (' Infinite number of Solutions.\n' ) ; 

22 end 

23 end 


The resuit of executing the above MATLAB script is: 

Enter the matrix A: [123; 456; 789] 

Enter the vector b: [1;3;5] 

Rank(A) = Rank([A b]) = 2 < 3 = #Col(A). 

Infinite number of Solutions. 

Enter the matrix A: [135; 246; 789] 

Enter the vector b: [1;1;1] 

Rank(A) = 2 ~= 3 = Rank([A b]). 

There is no solution. 

Enter the matrix A: [22-1; 121; -1-12] 

Enter the vector b: [2;4;1] 

Rank(A) = Rank([A b]) = 3 = #Col(A). 

There is a unique solution, given by: 

0.6667 
1.0000 
1.3333 

In Python, the built-in function sympy.Matrix is used to construet a 
matrix. The Matrix class has a method rref to compute the reduced row 
echelon from of the matrix. 


1 import sympy as smp 

2 A = smp.Matrix{[[2, 2, -1], [1, 2, 1], [-1, -1, 2]]) 

3 b = smp.Matrix{[[2], [4], [1]]) 

4 m, n = A.rows, A.cois 

5 r 1 = A.rank () 

6 C = A.copy() 

7 r2 = (C.row_join(b)). rank () 
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8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 


if rl != r2: # testing whether rank (A) ... 

not equal rank ( [A b] ) 

print ( 'Rank(A) = ' +str{rl) +' != ' +str(r2) +' = Rank([A ... 

b]) . ' ) 

print ('There is no solution.\n' ) ; # No solution in this case 

if rl == r2: # testing whether rank(A) = ... 

rank ( [A b]) 

if rl == n: # if yes, testing whether the ... 

rank equals n 

R = (A.row.join(b)).rref() # the reduced row ... 

echelon form of [A b] 

X = R[0][:, -1] # the solution is at the last ... 

column of the reduced 

# row echelon form 

print { 'Rank(A) = Rank([Ab]) = '+str(rl) +' = #Col(A).') 
print ('There is a unique solution, given by: ') ; ... 

print (x) ; 

#displaying the solution of the linear system 
else: # rank(A) = rank ( [A b]) < n 

print (' Rank(A) = Rank([A b]) = ' +str(rl) +' < ' ... 

+str(n) +' = #Col(A).' ) 
print (' Infinite number of Solutions.\n' ) 


By executing the code, the following results are shown: 


Rank(A) = Rank([A b]) = 3 = #Col(A). 
There is a unique solution, given by; 
Matrix([[2/3], [1], [4/3]]) 


1.3 Matrix Factorization Techniques 

Matrix factorization means to express a matrix A as a multiplication of two 
or more matrices, each is called a factor [34, 21]. That is, to write: 

A = Ai ■ A 2 ■ ■ ■ ■ ■ An 

In this section, three important matrix factorization techniques will be 
discussed; namely, the LU factorization, the QR factorization and the 
singular value decomposition (SVD). Then, the use of those factorization 
methods in solving linear Systems of equations will be discussed. 

Because cases of solving linear Systems with upper or lower triangular 
matrices will be encountered, this section will start by writing MATLAB and 
Python codes for solving such a linear system. 

1.3.1 The LU Factorization 

In this factorization, the matrix A is expressed as a multiplication of two 
matrices L and U, where L is an lower triangular matrix and U is an upper 
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triangular matrix. That is: 



( hi 

0 

0 

... 0 \ 


/uil 

Ul2 

Ml3 

U±ri'^ 


hi 

122 

0 

... 0 


0 

U22 

U23 

U 2 n 

A = L-U = 

hi 

h2 

^33 

... 0 


0 

0 

U33 

■ ■ • '^Sn 


\Jnl 

^n2 

^n3 

^nn j 


l 0 

0 

0 

Unn J 


where Ijj = 1 for j = 1,2,..., n. 

The function lu can be used for finding the L and U factors of matrix S. 
In MATLAB, this can be done as follows: 

» A = [4 -1 -1; -1 4 -1; -1 -1 4] 

A = 

4 -1 -1 

-1 4 -1 

-1 -1 4 


» [ L , u ] 

= lu(A) 


L = 

1.0000 

0 

0 

- 0.2500 

1.0000 

0 

- 0.2500 

- 0.3333 

1.0000 

U = 

4.0000 

- 1.0000 

- 1.0000 

0 

3.7500 

- 1.2500 

0 

0 

3.3333 


In Python, the function lu is located in the scipy.linalg sub-package 
and can be used to find the LU factors of matrix A. 

In [1]: import numpy as np, scipy.linalg as Ig 

In [2]: A = np.arrayC[[4, -1, -1], [-1, 4, -1], [-1, -1, 4]]) 

In [3]: P, L, U = Ig.lu(A) 

In [4]: print(’L = \n’, L, ’\nU = \n’, U) 

L = 


[[ 1 . 

0 . 

0 


] 

[- 0.25 

1 . 

0 . 


] 

[- 0.25 

- 0.33333333 

1 . 


]] 

U = 





[[ 4 . 

- 1 . 

-1 


] 

[ 0 . 

3.75 

- 1 . 

25 

] 

[ 0 . 

0 . 

3 . 

33333333 ]] 


However, python can compact both the L and U factors of matrix A using 
the function lu_factor. 
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In [5]; LU = Ig.lu_factor(A) 

In [6]: print(’LU = \n’, LU) 

LU = 

(array([[ 4. , -1. , -1• ] , 

[-0.25 ,3.75 , -1.25 ], 

[-0.25 , -0.33333333, 3.33333333]]), array([0, 1, 2], 

dtype=int32)) 

Now, the linear system 1.1 becomes: 


/1 

0 

0 

... o\ 


/uil 

Ul 2 

Ml3 

Uln^ 


( 


/h\ 

hi 

1 

0 

... 0 


0 

U22 

U23 

■ •• U 2 n 


X 2 


b2 

hi 

132 

1 

... 0 


0 

0 

U33 

'^Zn 


X 3 

= 

h 

\^nl 

ln2 

^nZ 

..: ij 


V 0 

0 

0 

'^nn } 


\XnJ 


\bn) 


( 1 . 6 ) 

The solution of the linear system 1.6 is found in three stages: 


1. First: we let y = Ux, that is 



Ull 

U\2 

Ml3 

Uln ' 


f xU 


0 

U22 

U23 

U2n 


X2 

y = 

0 

0 

U33 

• •■ ^Zn 


X3 


u 

0 

0 

'^nn J 


\Xn / 


Then, solving system 1.6 is equi valent to solving the linear system 

Ly^b 


2. Second: we solve the system Ly = b using the function SolveLower 
System.m to find y. 

3. Finally: we solve the linear system Ux = y using the back-substitution 
method, implemented by the MATLAB function SolveUpperSystem. 


Example 1.1 In this example, the LU-factors will be used to solve the linear 
system: 



In MATLAB, the following commands can be used: 


» A = [4 -1 -1; -1 4 -1; -1 -1 4] ; 
» b = [2; 2; 2] ; 

» [L, U] = lu(A) 
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L = 

1.0000 

0 

0 

- 0.2500 

1.0000 

0 

- 0.2500 

- 0.3333 

1.0000 

U = 

4.0000 

- 1.0000 

- 1.0000 


0 3.7500 -1.2500 

0 0 3.3333 

>> y = SolveLowerLinearSystem(L, b, 3) 

y = 

2.0000 

2.5000 

3.3333 

>> X = SolveUpperLinearSystemCU, y, 3) 

X = 

1.0000 
1.0000 
1.0000 

In Python, similar steps can be followed to solve the linear system Ax = b 
using the LU factors of matrix A. 

In [7]; y = lg.solve(L, b) 

In [8]; X = Ig.soIveCU, y) 

In [9]; print(’x = \n’, x) 

X = 

[[ 0.5] 

[ 0.5] 

[ 0.5]] 

Python has the LU solver Iu_soIve located in scipy.Iinalg sub-package. 
It receives the matrix LU obtained by applying the lu_solve function, to 
return the solution of the given linear system. 

In [10]; X = Ig.lu_solve(LU, b) 

In [11]; print(x) 

[[ 0.5] 

[ 0.5] 

[ 0.5]] 

The Python’s symbolic package sympy can also be used to find the LU 
factors of a matrix A. This can be done as follows: 

In [10]: import sympy as smp 

In [11]: A = smp.Matrix([[ 4 ■, -1., -!■], [-1-, 4-> 

[-1., -1., 4.]]) 

In [12]: LU = B.LUdecompositionO 
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In [13]: LU 
0ut[13]: 

(Matrix([ 

[ 1 , 0 , 0 ], 

[-0.25, 1, OJ, 

[-0.25, -0.333333333333333, IJJ), Matrix([ 

[ 4 . 0 , - 1 . 0 , - 1 . 0 ], 

[ 0, 3.75, -1.25], 

[ 0, 0, 3.33333333333333]]), []) 

In [ 14 ]: LU[0] 

Out[ 14 ]: 

Matrix([ 


[ 

1 , 

0 , 

0 ], 

[- 0 . 

.25, 

1 , 

0 ], 

[- 0 . 

.25, 

-0.333333333333333, 

1 ]]) 


In [15]: LU[1] 

0ut[15]: 

Matrix([ 

[ 4 . 0 , - 1 . 0 , - 1 . 0 ], 

[ 0, 3.75, -1.25], 

[ 0, 0, 3.33333333333333]]) 

The symbolic package sympy can be also used to solve a linear system, using 
the LU factors. 

In [16]: h = [[2.0], [2.0], [2.0]] 

In [17]: A.LUSolve(b) 

Out[17] : 

Matrix([ 

[ 1 . 0 ], 

[ 1 . 0 ], 

[1 . 0 ]]) 

1.3.2 The QR Factorization 

In this type of factorization, the matrix A is expressed as a multiplication of 
two matrices Q and R. The matrix Q is orthogonal (its columns constitute an 
orthonormal set) and the matrix R is an upper triangular. 

From the elementary linear algebra, an orthogonal matrix satisfies the 
following two conditions: 

1. Q~^ = , and 

2. if Q= [qiq 2 ---qn], then, 

/ N T f 1 * = J 

= i*, 
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In MATLAB, the function qr can be used for finding the QR factors of a 
matrix A. The command is as follows: 

» A = [4 -1 -1; -1 4 -1; -1 -1 4] ; 

» [Q, R] = qr(A) 

Q = 

-0.9428 -0.1421 0.3015 

0.2357 -0.9239 0.3015 

0.2357 0.3553 0.9045 

R = 

-4.2426 1.6499 1.6499 

0 -3.9087 2.4873 

0 0 3.0151 

In Python, the function qr located in scipy.linalg can be used to find 

the QR factors of matrix A. 

In [18]; Q, R = Ig.qr(A) 

In [19]; print(’Q = \n’, Q, ’\nR =\n’, R) 

Q = 

[[-0.94280904 -0.14213381 0.30151134] 

[ 0.23570226 -0.92386977 0.30151134] 

[ 0.23570226 0.35533453 0.90453403]] 

R = 

[[-4.24264069 1.64991582 1.64991582] 

[ 0. -3.9086798 2.48734169] 

[ 0. 0. 3.01511345]] 

The symbolic Python can also be used to find the QR-factors of matrix A 

In [20]; QR = A. QRdecompositionO 
In [21]; QR 
Out [21] ; 

(Matrix([ 

[0.942809041582063, 0.14213381090374, 0.301511344577764], 

[-0.235702260395516, 0.923869770874312, 0.301511344577764], 

[-0.235702260395516, -0.355334527259351, 0.904534033733291]]), 
Matrix([[4.24264068711928, -1.64991582276861, -1.64991582276861], 
[ 0, 3.90867979985286, -2.48734169081546], 

[ 0, 0, 3.01511344577764]])) 

In [22]; QR[0] 

Out [22] ; 

Matrix([ 

[0.942809041582063, 0.14213381090374, 0.301511344577764], 

[-0.235702260395516, 0.923869770874312, 0.301511344577764], 
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[-0.235702260395516, -0.355334527259351, 0.904534033733291]]) 

In [23]: QR[1] 

Out [23] : 

Matrix([ 

[4.24264068711928, -1.64991582276861, -1.64991582276861], 

[ 0, 3.90867979985286, -2.48734169081546], 

[ 0, 0, 3.01511344577764]]) 

To solve the linear system 1.1, using the QR factorization technique the 
following steps can be used: 

1. Finding the QR factors of matrix A and rewrite the system Ax = b as 
Q ■ Rx = b. 

2. multiplying the two sides of equation Q ■ Rx — b hy , giving: 

Q^QRx = Q^b ^ Rx = Q^b 


3. solving the upper triangular system Rx = y, where y = Q^b using 
the backward substitution method, implemented by the function 
SolveUpperLinearSystem.m. 

Example 1.2 In this example, the QR factors will be used to solve the linear 
system: 



» A = [4, -1, -1; -1, 4, -1; -1, -1, 4] ; 

» b = [2; 2; 2] ; 

» [Q, R] = qr(A) ; 

» y = Q’*b 

y = 

-0.9428 

-1.4213 

3.0151 

>> X = SolveUpperLinearSystem(R, y, 3) 

X = 

1.0000 
1.0000 
1.0000 

Python can be used to solve the above linear system, using the QR factors 
as follows. 

In [24]: import numpy as np, scipy.Iinalg as Ig 

In [25]: A = np.arrayC[[4, -1, -1], [-1, 4, -1], [-1, -1, 4]]) 
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In [26]; b = np.arrayC[[2.0],[2.0],[2.0]] ) 

In [27]; Q, R = Ig.qr(A) 

In [28]; y = np.matmulCQ.T, b) 

In [29]; x = Ig.soIve(R, y) 

In [30]; print(’x = \n’, x) 

X = 

[[ 1 . 0 ] 

[ 1 . 0 ] 

[ 1 . 0 ]] 

Another method is to use the symbolic package: 

In [31]: import sympy as smp 

In [32]: A = smp.Matrix([[ 4 .0, -1.0, -1.0], [-1.0, ~l-0], 

[- 1 . 0 , - 1 . 0 , 4 . 0 ]]) 

In [33]: b = smp. Matrix ([[2.0], [2.0], [2.0]]) 

In [ 34 ]: X = A.QRsolve(b) 

In [35]: x 
0ut[35]: 

Matrix([ 

[ 1 . 0 ], 

[ 1 . 0 ], 

[1 . 0 ]]) 


1.3.3 The Singular Value Decomposition (SVD) 

In the svd decomposition, the matrix A is expressed as a multiplication of 
three matrices U, S and V'^, that is; 

A = U-S-V^, 

where both U and V are unitary matrices and S” is a diagonal matrix. 

The columns of matrix U are the eigenvectors of the matrix AA'^. The 
columns of matrix V are the eigenvectors of the matrix A. The diagonal 
elements of the matrix S are the squares of the eigenvalues of matrix A. 

If are the columns of U, and {v\,V 2 t. .. ,Vn} the columns of V, 

then they satisfy the following conditions; 

1. U~^ = U* and V~^ =V*. If U and V have real entries, then U~^ = 
and V~^ = V'^. 

2. the columns of U and V are orthonormal sets; 


{Ui,Uj) = u* -Uj = 


1 i = j 

0 ii^j 


and 


{Vi,Vj) = V*-Vj = 


1 i = j 

0 ii^j 
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In MATLAB, the command svd is used to find the svd components of a 
matrix A. 

» A = [4 -1 -1; -1 4 -1; -1 -1 4] ; 

» [U, S, V] = svd(A) 

U = 

0.0000 -0.8165 -0.5774 

-0.7071 0.4082 -0.5774 

0.7071 0.4082 -0.5774 

S = 

5 0 0 

0 5 0 

0 0 2 

V = 

0 -0.8165 -0.5774 

-0.7071 0.4082 -0.5774 

0.7071 0.4082 -0.5774 

In Python, the function svd is used to find the svd decomposition of 
matrix A. 

In [36]; U, S, V = Ig.svd(A) 

In [37]; print(’U = \n’, U, ’\nS = \n’, S, ’\nV = \n’, V) 

U = 

[[ 2.69618916e-17 -8.16496581e-01 -5.77350269e-01] 

[ -7.07106781e-01 4.08248290e-01 -5.77350269e-01] 

[ 7.07106781e-01 4.08248290e-01 -5.77350269e-01]] 

S = 

[ 5. 5. 2.] 

V = 

[[ 0. -0.70710678 0.70710678] 

[-0.81649658 0.40824829 0.40824829] 

[-0.57735027 -0.57735027 -0.57735027]] 

Now, solving system Ax = b is equivalent to finding the solution of the linear 
System 

USV'^x = b, (1.7) 

hence, multiplying the two sides of Equation (1.7) by V ■ S~^U^, gives, 

x = V-S-^U'^b 

Example 1.3 The svd will be used to solve the linear system: 
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» X = V*inv(S)*U’*b 

X = 

1.0000 
1.0000 
1.0000 

In Python, the linear system is solved with the svd components as follows: 

In [38]; x = Ig.solveCV, Ig.solve(np.diag(S), lg.solve(U,b))) 

In [39]: print(’x = \n’, x) 

X = 

[[ 0.5] 

[ 0.5] 

[ 0.5]] 
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Solving Linear Systems with Iterative and 
Least Squares Methods 


Abstract 

The direct methods for solving a linear system Ax = b are to try to find 
the exact solution of the linear system, by inverting the matrix A directly or 
indirectly. The iterative methods aim at finding an approximate solution of 
the linear system by finding a sequence of vectors that is converging to the 
exact solution. In the case that the linear system does not have a solution, 
the problem turns into a least squares problem. 

This chapter aims to find approximate and least squared Solutions of linear 
Systems. It is divided into three sections. In the first section, basic concepts 
such as error norm and convergence of vector sequences are introduced. Then, 
in the second section three iterative methods for finding approximate Solutions 
of linear Systems of equations are discussed and implemented in MATLAB® 
and Python. When a linear system does not have a solution, the problem 
turns into searching for a least squares solution that minimizes the error norm. 
Examples of least squares problems and best approximations of functions by 
polynomials are discussed and implemented in MATLAB and Python, in the 
third section. 


2.1 Mathematical Backgrounds 

This section present the concepts of vectors norms and the convergence of 
sequences in a vector space MA. 

2.1.1 Convergent Seqnences and Canchi’s Convergence 

Let a be any point in R, and e G R“*~ be any positive real number. The e- 
neighbourhood of a (denoted by J\f{a,e)) is the set of all points a; € R that lie 
in the open interval (a —e,a + e). That is, J\f{a,e) = {a; G R : |a; — a| < e} [46]. 

A sequence of real numbers {a„}[][LQ = « 0 , 01 , 02 ,... is said to converge 
to o G R, if for any choice of e > 0, J\f{a,e) contains an infinite sub-sequence 
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= o,N,CLN+i,aN+ 2 ,--- of the sequence {an}^o- To express the con- 
vergence of the sequence {onj^o ® mathematically, we write 

|a„ — a\<e, \/n> N 

For example, the sequence 

n , 

1 as n —>■ oo 


n+^}n=o 

because, if we make any choice for £ > 0, then 


n 


-1 


-1 


n+ 1 


n+ 1 

and therefore, if we let 

N = 

then for any integer n> N we find 

n 


1 1 , 1-e 

-- <e=^n>-1=-, 

n+1 e e 


1 — e 


n+1 


-1 


< e. 


Definitiori 2.1 (Convergence of a sequence [46]) Let {an} be a real- 
valued sequence. We say that the sequence {an} converges to a real number a, 
if for any e > 0, there exists iV G N such that: 

\an — a\<e, Vn > N. 

In this case, we say that a is the limit of sequence {an}, and we write 

lim Un = a 

n—^oo 

It is worthy to notice that if {on} is a convergent sequence (to some limit 
a), then, 

|ao — oil > \a\ — a2 \ > ■■■ > |an — «n+il > • • • 

and 

{\an — fln+i 1} —>-0, as n —> oo 

For example, in the sequence the corresponding sequence 

{\an- an+l\} = (n+i)\n+2) > 

1111 
2 ’ 6 12’20 

Definitiori 2.2 A sequence {an} is Cauchy if for any £ > 0, there exists N G 
N, such that 

\an — flml < £,for ali n,m> N. 


Because R is complete, any convergent sequence is Cauchy’s sequence and 


vice-versa. 
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2.1.2 Vector Norm 

If a; is a vector with components xi,X 2 , ■ ■ ■ ,Xn, then its p*^-norm is defined 
by: 

ii^iip= ={\x^\p+\x2\^+...+Kn'^ 

It satisfies |la:|ji > ||a;|j 2 > •■■ > ||ic||oo- The norm ||£c ||2 gives the classical 
Euclidean distance: 

||a:|| 2 = ^Jxl + xl + .-. + xl 

The Python function norm (located in numpy. linalg library) receives a vector 
X and an integer p or np.inf, and returns ||a:||p- 

In [27]; x = np.array([l, -1, 2, -2, 3, -3]) 

In [28]; from numpy.linalg import norm 

In [29]; nl, n2, n3 = norm(x, 1), norm(x, 2), norm(x, np.inf) 

In [30]; printCnl, n2, n3) 

12.0 

5.29150262213 

3.0 

2.1.3 Convergent Seqnences of Vectors 

The distance in a vector space can be defined by using any norm. If we have 
two vectors x and y both are in R”, then the distance between x and y can be 
defined by |ja;-y||i, ||a:-y|| 2 , ... or ||a;-y||oo. Usually, ||a;-y ||2 or ||a;-y||oo 
are used as values for the distance between two vectors x and y. 

Definition 2.3 (Convergence of seqnences in vector spaces [4]) Let 

x^^\x^^\... be a sequence of vectors in R”. We say that the sequence 
converges to a vector x € R*^, iffor any e > 0, there exists iV G N, 

such that 

||3.(»«) _ jp|| ^ Trn>N. 

Definition 2.4 (Cauchy’s Convergence of seqnences in vector spaces 
[45]) Let ... be a sequence of vectors in R"'. We say that the sequence 

is Cauchy convergent, iffor any £ > 0, there exists 7V G N, such that 

< e, for all m,s> N. 


2.2 The Iterative Methods 

The iterative methods for solving linear Systems of equations, is to start from 
a given initial guess and generate a sequence of vectors ... that 
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converge to the solution of the linear system x — x*. The generated sequence 
stops at some vector x^^'> that satisfies: 

where || ||. is some norm in R” and e > 0 is a given tolerance. Then, the 
solution of the linear system x* is approximated by x^^\ that is x* « 

Generally, there are two kinds of iterative methods [47]: 

1. stationary iterative methods: at an iteration k the iterative method 

computes x^^') from without referring to the previous history. This 

class of methods includes the Jacobi, Gauss-Seidel and the relaxation 
methods. 

2. non-stationary iterative methods: at iteration fc, the iterative method 

refers to the whole history x^^\x^^'),... for the computation of 

x^^\ This class of methods includes the conjugate gradient and the 
GMRES subspace methods. 

In this section, three stationary iterative methods for solving linear system 
will be discussed. The methods include the Jacobi, Gauss-Seidel and relaxation 
methods. 

2.2.1 The General Idea 

Given the linear system Ax = b, e > 0 and an initial point The goal is 
to generate a sequence of vectors in R"" 

0 .( 0 ) ™( 1 ) 0 .( 2 ) 

nXj' ^ ^ dL/ ^ • 

that converges to the solution of the linear system Ax = b. That is 

lim = x* 

k^oo 

The generation of the sequence stops at some iteration s, with 

|ja;(*)-a:("-i)|| <£ 

In a stationary iterative method, the matrix A is expressed as a sum of 
two matrices S and T, that is A = S + T, where S is an invertible matrix. 
Then, the linear system Ax — b is replaced by (^-l-Tja: = 5 or 

Sx = b — Tx 

Since S is invertible, we multiply the two sides by 5"“^ and obtain a linear 
System: 

X = Bx + c 

where B = —S~^T and c = S~^b. 
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The solutiori of the above linear system is a fixed point x*, where 

X* = Bx*+c 

The fixed point is approached iteratively, starting from the given initial 
point x^^\ by using the iterative relationship: 


Through the whole of this section, we assume that we write matrix A is a 
sum of three matrices L, D and U, where: 


/ 0 

0 

0 

0 

0 

02,1 

0 

0 

0 

0 

03,1 

03,2 

0 

0 

0 

an-2,1 

an-2,2 

an-2,3 ••• 

0 

0 

On-l,! 

On-l,2 


^n—l,n—2 

0 

V On,l 

On,2 

0^n,3 

0^n,n—2 

^n,n 



and 


' Ol,l 

0 

0 

0 

0 


0 

02,2 

0 

0 

0 

0 

0 

0 

03,3 

0 

0 

0 

0 

0 

0 

On—2,n- 

-2 0 

0 

0 

0 

0 

0 

On—l,n- 

-1 0 

V 0 

0 

0 

0 

0 

Q'n,n / 

/0 

Ol,2 

Ol,3 

Oi^n_2 

Ol,n—1 

0-1,n 

0 

0 

a2 ,3 

02,n-2 

02,n-l 

0^2,n 

0 

0 

0 

03,n-2 

03,n-l 

0-3,n 

0 

0 

0 

0 

On —2,n —1 

^n—2,n 

0 

0 

0 

0 

0 

^n—l,n 

\0 

0 

0 

0 

0 

0 / 


In MATLAB, matrices L,D and U can be obtained by using the com- 
mands: 


>> D = diag(diag(A)) ; 
>> L = tril(A) - D ; 


>> U = triu(A) - D; 

In Python, the tril, triu are implemented in the scipy.linalg pack- 
age, and diag is implemented in both scipy and numpy. They can be obtained 
through the following commands: 
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In [1]; import scipy as sp, numpy as np 

In [2]: A = np.array([[4, 1, -1], [-1, 5, 1], [0, -1, 4]]) 
In [3]; print(’L = \n’, sp.Iinalg.tril(A), ’\nU = \n’, 
sp.Iinalg.triu(A), ... 

’\nD = \n’, np.diagCnp.diag(A))) 

L = 

[[ 4 0 0] 

[-1 5 0] 

[0-1 4]] 

U = 

[[41 - 1 ] 

[051] 

[ 0 0 4]] 

D = 

[[4 0 0] 

[0 5 0] 

[0 0 4]] 

2.2.2 The Jacobi Method 

Given the linear system of equations: 

aiiXi+ai2X2 +■■■ + ainXn = h 
a 2 lXi+a 22 X 2 +■■■ + a 2 nXn = h 


anXi + ai2X2 + ■ ■ ■ + ainXn = h ( 2 . 1 ) 


^n2X2 “t“ . ■ . “t“ annXn — 
From the above equation, follows that: 

xi = — {bi-ai2X2-■■■-aniXn) 

aii 

X2 = - ib2 - a2lXi - a23X3 - ■ ■ ■ - anlXn) 

«22 


Xi — {bi UiiXi ... an—iXi—i aii-^iXi-\-i ... ainXn) ( 2 . 2 ) 


Xyi — {bn anlXi . . . ann—lXn—1^ 

^nn 

The Jacobi method is an iterative method, which starts from an initial 
guess for the solution • ■ .,Xn '^]’^. Then, the solution in iteration k is 
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used to find an approximation for the System solution in iteration k+1. This 
is done as follows: 

(fe+i) 1 /, (fe) (fe)\ 

X { ' = - 01-012X2 " 

oii V / 

(fe+i) 1 /, (fe) (fe) (fe)\ 

X2 =-I 02 - a 2 ixl - 023X3 a„ix„ 

022 ^ ^ 

(fe + 1) ^ /l (fc) (fc) (fc) /O Q\ 

x^ = - I Oi-OjiXj - .-Oii_iX^_\-Oii+ixi_,_j - .-OinXn I ( 2 . 3 ) 


— 


(u (fc) (fe) ^ 

(^On - OnlX) - ... - ann-lX\_i J 
(k-hl) 

Generally, the solution in iteration x) ^ can be written in the form: 


Jk+i) 


1 


= — bi- ^ aijX^p ,z = l,...,n 


(2.4) 


j = l,2,ti 

The Jacobi iteration stops when, 


llxl^^+i) -x(^)|| < 


for some arbitrary e > 0. 


Example 2.1 Write the first three iterations of the Jacobi method, for the linear 
System: 



starting from the zeros vector 



Solution: 



1 

2 


^2 — 

1 

5 


X3 — 

1 

4 



We write: 



30 


Solving Linear Systems with Iterative and Least Squares Methods 


1. First iteration k = 0: 


Jl) 

- r 

(-1+4°^-4°\ 

) = l(_l + 0-0) = 

1 

~2 

Jl) 

J.2 

= r 

(^l + 2a;4 +4°^) 

= i(l + 2(0) + 0) = 

1 

5 

M) 

X 3 

- / 

(^3-a;4+2x^°^) 

= i(3_0 + 2(0)) = 

3 

4 


2. Second iteration k=l: 


.(2) 


.(2) 


.(2) 




( 

i (l + 2a;4+4^^) 
\ (3-a;4+2x4) 


^1/ 1_3N^ _31 

2 V 5 4/ 40 

1 ^ -1 3\ 3 

1 / -1 39 


3. Third iteration k = 2: 


/3) 


/3) 


/3) 




i(l + 2x4 + xf 


J(3-4"42xf) 


_ 1 / 3 39\ _ 73 

“ 2 !^” ”^^”40/ 

5 V 40 80 y 200 

1 / 31 „ 3 \ 163 

“ 4 (^“40”^^^) “ 160 


Example 2.2 The Jacobi method will be applied for solving the linear system: 



1 function x = JacobiSolve(A, b, Eps) 

2 n = length (b) ; 

3 xO = zeros (3, 1) ; 

4 X = ones (size (xO)) ; 

5 while norm(x-x0, inf) > Eps 

6 xO = X ; 

7 for i = 1 : n 

8 X {i) = b (i) ; 

9 for j = 1 : n 

10 if j ^ i 

11 x(i) = x(i) - A{i, j)*x0{j) ; 

12 end 

13 end 

14 x(i) = x(i) / A(i, i) ; 

15 end 
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» A = [-5 1 -2; 1 6 3; 2 -1 -4] ; 

» b = [13; 1; -1] ; 

>> JacobiSolveLinSystemCA, b) 

- 2.0000 
1.0000 
- 1.0000 

Using Python, the function JacobiSolve has the following code: 


1 def JacobiSolve (A, b, Eps): 

2 import numpy as np 

3 n = len (b) 

4 xO, X = np.zeros((n, 1), 'float')/ np.ones((n, 1), 'float') 

5 while np.linalg.norm(x-xO, np.inf) > Eps: 

6 xO = X.copy() 

7 for i in range(n) : 

8 X [i] = b [i] 

9 for j in range(n) : 

10 if j ! = i: 

11 x[i] -= A[i] [ j] *x0 [ j] 

12 X [i] /= A[i] [i] 

13 return x 


By calling the function JacobiSolve to solve the given linear system, we obtain: 

In [3]; A = np.arrayC[[-5, 1, -2], [1, 6, 3], [2, -1, -4]]) 

In [4]; b = np. arrayC [ [13] , [1], [-1]]) 

In [5]; X = JacobiSolve(A, b, Eps) 

In [6]; print(’x = \n’, x) 

Out [6] ; 

X = 

[[- 2 . ], 

[ 1.000000021, 

[- 1 . 00000002 ]] 


2.2.3 The Jacobi Method in the Matrix Form 

Matrix A can be expressed as: 


A = L + D + U, 

therefore, the linear system Ax = b can be written as: 

{L + D + U)x = b 

The Jacobi method chooses S = D and T = L + U. It is worthy to notice 
that no diagonal element in D can be 0. That is da + 0, for all i = 1,...,n. 
The Jacobi method is of the form: 

where, B = —D~^{L + U) and c = D~^b. 
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The following MATLAB code implements the Jacobi method in the matrix 
form: 


1 

function [x, Iters] = JacobiIter_VectorForm(A, b, Eps, xO) 

2 

D = diag (diag (A)) ; 


3 

B = -D\(A-D) ; 


4 

c = D\b ; 


5 

Iters = 1 ; 


6 

xl = B*xO + c ; 


7 

while norm(xl-xO, inf) > Eps 


8 

xO = xl ; 


9 

xl = B*xO + c ; 


10 

Iters = Iters + 1 ; 


11 

end 


12 

X = X 1 ; 



Applying the above code to the linear system in 

Example 2.2: 

» 

A = [-5 1 -2; 1 6 3; 2 -1 -4] ; 


» 

b = [13; 1; -1] ; 


» 

[x, Iters] = JacobiIter_VectorForm(A, b, 

Eps, xO) 

X 



-2 

.0000 


1 . 

0000 


-1 

.0000 


Iters = 


26 




The Python code for the function JacobiIter_VectorForm is: 

1 

def Jacobilter.VectorForm(A, b, xO, Eps): 


2 

import numpy as np, scipy as sp 


3 

D = np.diag(np.diag(A)) 


4 

B = -sp.linalg.solve(D, A-D) 


5 

c = sp.linalg.solve(D, b) 


6 

Iters = 1 


7 

X = np.matmul(B, xO)+c 


8 

while np.linalg.norm(x-xO) > Eps: 


9 

xO = X .copy() 


10 

X = np.matmul(B, xO)+c 


11 

Iters += 1 


12 

return x, Iters 



From the IPython console, the following commands can be used to solve the 
linear system: 


In [7]; X, Iterations = JacobiSolve_VectorForm(A, b, xO, Eps) 
In [8]; print(’Iterations = Iterations, ’\nx = \n’, x) 
Iterations = 24 

X = 

[[ - 2 . 


] 
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[ 1 . 00000002 ] 
[- 1 . 00000002 ]] 


2.2.3.1 The Gauss-Seidel Iterative Method 

The Gauss-Seidel method is close to the Jacobi method, but it benefits from 
the recently updated variables in iteration fc-l-l to update 

whereas the other variables ■ ■ • ,x^n'^ are taken from the previous 
iteration k. The Gauss-Seidel is of the form: 


(fe+i) 

ai^ixl 

(fe-l-i) , (fe+i) 

a2,lXi +a2^2X2 


h (^) (^) 

0l - ai2a;2 - ... - Onl^n 

7 (fe) (k) 

02 - a 2 ZXl - ... - anlXn 


{k+l) 

ai,ixl 


-Qi^iX 


(k+l) 


h ('=) 


(2.5) 


(fe+l) , , (*:+!) 

an,ix\ ' + .. . + ai,iXl 

From equation (2.5), 


— br 


.(*+l) _ 

1 — 

1 

(bi-ai2xi^^ - 

Oii 

.(fe+1) 

1 

(h n + A 

2 — 

022 

(02 — a2ia;^ 




(k) 


) 


Sk + l) 


— [bi - anxf^^'* - ... - aii-ixf_X^^ - au+ixfi^-^ ( 2 . 6 ) 

CLri ^ 


{k) 

din^n 


Sk + l) 


dnn ^ ' 


The following MATLAB function implements the Gauss-Seidel method in the 
equation form: 


1 function [x, Iters] = GaussSeidel(A, b, Eps, xO) 

2 Iters = 1 ; 

3 xl = zeros (size (xO)) ; n = length(b) ; 

4 for i = 1 : n 

5 xl(i) = b(i) ; 

6 for j = 1 : i-1 

7 xl{i) = xl (i)-A{i, j) *xl ( j) ; 

8 end 

9 for j = i+1 : 


n 
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10 xl(i) = xl (i) - A{i, j)*xO(j) ; 

11 end 

12 xl (i) = xl(i)/A(i,i) ; 

13 end 

14 while norm(xl-xO, inf) > Eps 

15 xO = xl ; 

16 for i = 1 : n 

17 xl (i) = b (i) ; 

18 for j = 1 : i-1 

19 xl(i) = xl (i)-A {i, j ) *xl { j ) ; 

20 end 

21 for j = i + 1 : n 

22 xl{i) = xl{i) - A(i, j)*xO{j) ; 

23 end 

24 xl (i) = xl (i)/A {i, i) ; 

25 end 

26 Iters = Iters + 1 ; 

27 end 

28 X = xl ; 


Applying the above code to the linear Systems in examples 2.1 and 2.2: 


>> Eps = le-8 ; xO = [0;0;0] ; 

» A = [2 -1 1; -2 5 -1; 1 -2 4] ; b = [-1;1;3] ; 
>> [x, Iters] = GaussSeidel(A, b, Eps, xO) 

X = 

- 1.0000 

0 

1.0000 
Iters = 


11 

» A = [-5 1 -2; 1 6 3; 2 -1 -4] ; b = [13; 1; -1] ; 
>> [x, Iters] = GaussSeidelCA, b, Eps, xO) 

X = 


- 2.0000 
1.0000 
- 1.0000 


Iters = 


15 


The python code for the function GaussSeidel is: 


1 import numpy as np 

2 def GaussSeidelSolve(A, b, xO, Eps): 

3 n = len (b) 

4 X = np.ones((n, 1), 'float') 

5 Iterations = 1 

6 while np.linalg.norm(x-xO, np.inf) > Eps: 

7 xO = x.copy() 

8 for i in range(n) : 

9 X [i] = b [i] 
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10 for j in range(i) : 

11 x[i] -= A[i] [ j] *x[ j] 

12 for j in range(i + l, n) : 

13 x[i] -= A[i] [ j] *x0 [ j] 

14 x[i] /= A[i] [i] 

15 Iterations += 1 

16 return x, Iterations 

17 

18 A = np.array{[[-5, 1, -2], [1, 6, 3], [2, -1, -4]]) 

19 b = np.array{[[13], [1], [-1]]) 

20 xO = np.zeros((3, 1), 'float') 

21 Eps = le-8 

22 X, Iterations = GaussSeidelSolve(A, b, xO, Eps) 

23 print ('Iterations = Iterations, '\nx = \n', x) 


By executing the above code we obtain: 
Iterations = 16 

X = 

[[- 2 .] 

[ 1 .] 

[- 1 .]] 


2.2.4 The Gauss-Seidel Method in the Vector Form 

The Gauss-Seidel method chooses S = L + D and T = U. The Jacobi method 
is of the form: 

where, B = —{L + D)~^U and c= {L + D)~^b. 

The following MATLAB code implements the Gauss-Seidel method in the 
matrix form: 


1 

function [x, Iters] = 

GaussSeidellter(A, b, Eps, xO) 

2 

LD = tril(A) ; U = A - 

LD ; 

3 

B = -LD\U ; c = LD\b ; 


4 

Iters = 1 


5 

xl = B*xO + c ; 


6 

while norm(xl-xO, inf) 

> Eps 

7 

xO = xl ; 


8 

xl = B * xO + C ; 


9 

Iters = Iters + 1 

; 

10 

end 


11 

x = x 1 ; 



Applying the above code to examples 2.1 and 2.2: 


>> Eps = le-8 ; xO = [0;0;0] ; 

» A = [2 -1 1; -2 5 -1; 1 -2 4] ; b = [-1;1;3] ; 
>> [x, Iters] = GaussSeidellter(B, b, Eps, xO) 


X 
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- 1.0000 

0 

1.0000 
Iters = 

11 

» A = [-5 1 -2; 1 6 3; 2 -1 -4] ; b = [13; 1; -1] ; 
>> [x, Iters] = GaussSeidellter (A, b, Eps, xO) 

X = 

- 2.0000 
1.0000 
-1.0000 
Iters = 

15 


The function GaussSeidellter can be implemented in python as: 


1 

def GaussSeidellter(A, b, Eps, xO): 



2 

import numpy as np, scipy as sp 



3 

LD = sp.linalg.tril(A) 



4 

U = A - LD 



5 

B = -sp.linalg.solve(LD, U) 



6 

c = sp.linalg.solve(LD, b) 



7 

Iters = 1 



8 

X = np,matmul(B, xO) + c 



9 

while np.linalg.norm(x-xO, np.inf) 

> 

Eps : 

10 

xO = X .copy() 



11 

X = np.matmul(B, xO) + c 



12 

Iters += 1 



13 

return x, Iters 



14 




15 

A = np. array ( [ [-5, 1, -2], [1, 6, 3], 

[2 

-1, -4]]) 

16 

b = np.arrayi [[13], [1], [-1]]) 



17 

xO = np.zeros((3, 1), 'float') 



18 

Eps = le-8 



19 

X, Iterations = GaussSeidellter(A, b. 

xO 

Eps) 

20 

print ('Iterations = ', Iterations, '\n 

X 

= \n', X) 


The resuit after executing the above code is: 


Iterations = 15 

X = 

[[- 2.1 
[ 1.1 
[- 1 . 1 ] 


2.2.5 The Relaxation Methods 

A relaxation method is an iterative method for solving linear Systems based 
on Gauss-Seidel iteration and involves a parameter 0 < w < 2 with the purpose 
of accelerating the convergente rate of the iterative method to the solution of 
the linear system. 
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Multiplying the linear system Ax = bhy w and replacing Ahy U + D + L 
gives 


{wU + ujD + ijjL)x = ujb^ {ojU + (1 — {l — uj))D + ujL)x = ivb 
from which, 

{D + u;L)x = {{l — u})D — ujU)x + ujb 

The iterative method is obtained by replacing x at the left-hand side by 
and by x^^'^ at the right-hand side, giving the formula 

{D - = ((1 - w)D - ioU) x^^^ + ujb (2.7) 

When w < 1, the iterative method in (2.7) is under rela:xation method. 
When u) = 1 the method is identical to Gauss-Seidel iteration, and when uj > 1 
(2.7) is over-relaxation method. In the latest case (cu > 1), the resulting 
method is called the successive over-relaxation method and is abbrevi- 
ated as SOR. 

By writting = D-ojL, = (l-w)D + wi7, = S~^Tu and = 
S~^Lob, the SOR iterative method is: 

x^’^+^'> (2.8) 

The MATLAB function SOR.m receives a matrix A, a vector b, an initial 
starting vector a;o, a real value lo and a tolerance e, and returns an approximate 
solution of the system Ax = b within the given tolerance together with the 
number of iterations. 


1 

function [x, Iters] = S0R(A, b, w, 

xO, Eps) 

2 

D = diag (diag (A)) ; 


3 

L = tril(A) - D ; U = A - (L+D) 

; 

4 

B = -(D+w*L)\({w-1)*D+w*U) ; c 

= (D+w*L)\(w*b) ; 

5 

Iters = 1 ; 


6 

X = B*xO + c ; 


7 

while norm(x-xO, inf) > Eps 


8 

X 

II 

o 

X 


9 

X = B * xO + c ; 


10 

Iters = Iters + 1 ; 


11 

end 


12 

end 



Applying the above code to examples 2.1 and 2.2: 

» A = [2 -1 1; -2 5 -1; 1 -2 4] ; b = [-1;1;3] ; 
>> xO = [1.; 1.; 1.]; w = 1.25; Eps = le-8 ; 

>> [x, Iters] = S0R(A, b, w, xO, Eps) 


X = 

- 1.0000 
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0.0000 
1.0000 

» A = [-5 1 -2; 1 6 3; 2 -1 -4] ; b = [13; 1; -1] ; 
>> [x, Iters] = S0R(A, b, w, xO, Eps) 

X = 

- 2.0000 
1.0000 
- 1.0000 


Iters = 
33 


The Python code SORIter.py applies the SOR to solve the linear Systems 
in examples 2.1 and 2.2. 


1 

import numpy as np 


2 

from scipy.linalg import tril, solve 


3 

def SOR(A, b, w, xO, Eps): 


4 

D = np.diag(np.diag(A)) 


5 

L = tril(A) - D 


6 

U = A - (L+D) 


7 

B = solve{-{D+w*L), (w-l)*D+w*U) 


8 

c = solve((D+w*L), w*b) 


9 

Iters = 1 


10 

X = np.matmul(B, xO) + c 


11 

while np.linalg.norm(x-xO, np.inf) 

> Eps : 

12 

xO = X .copy() 


13 

X = np.matmul(B, xO) + c 


14 

Iters += 1 


15 

return x, Iters 


16 



17 

print (’Solving the first linear system: 

' ) 

18 

A = np.array{ [ [2, -1, 1], [-2, 5, -1]/ 

[1, -2, 4]]) 

19 

b = np.array([ [-1] , [1], [3]]) 


20 

W;. xO = 1,25, np.zeros((3, 1), 'float') 


21 

Eps = le-8 


22 

X, Iterations = SOR(A, b, w, xO, Eps) 


23 

print ( 'x = ' \nIterations = ' , 

Iterations) 

24 



25 

print (’Solving the second linear system:') 

26 

A = np.array( [ [-5, 1, -2], [1, 6, 3], | 

2, -1, -4]]) 

27 

b = np . array ([ [13] , [1], [-1]]) 


28 

X, Iterations = SOR(A, b, w, xO, Eps) 


29 

print ( 'x = \n', x, ' \nIterations = 

Iterations) 


Executing the above code gives the following results: 


runfileC’D;/PyFiles/SORIter.py’, wdir=’D;/PyFiles’) 
Solving the first linear system: 

X = 

[[-l.OOOOOOOOe+00] 
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[-1.94732286e-10] 

[ l.OOOOOOOOe+00]] 

Iterations = 16 

Solving the second linear system; 

X = 

[[- 2 .] 

[ 1 .] 

[- 1 .]] 

Iterations = 32 

The Gauss Seidel method solved Example 2.2 in 16 iterations, and the 
SOR method with w = 1.25 solved the example in 33 iterations. From this 
example, it is ciear that the SOR is not guaranteed to converge faster than 
Gauss-Seidel method if the parameter uj is not selected carefully. The selection 
of the relaxation parameter lo plays a key role in the convergence rate of the 
SOR. Hence when the optimal value of the parameter is selected, the SOR 
method achieves the best convergence rate. 

In the following example, 15 values in [0.1,1.5] are selected for parameter 
w and the corresponding numbers of iterations are computed. Then the values 
of w are plotted against the corresponding numbers of iterations. 


1 

W = np.arange{0.1, 1.6, 0.1) 


2 

Iterations = np.zeros.like(W) 


3 

for j in range ( len (W)): 


4 

X, Iters = SOR(A, b, W[j], xO, Eps) 


5 

Iterations[j] = Iters 


6 

import matplotlib.pyplot as plt 


7 

plt.figure(1) 


8 

plt.plot(W, Iterations, mark.er= ' s ' , color= ' purple 

lw=3) 

9 

plt.xticks(np.arange(0.1, 1.6, 0.2), fontweight = 

'bold' ) 

10 

plt.yticks(np.arange(25, 225, 25), fontweight = ' 

bold' ) 

11 

plt.grid(True, ls=':') 


12 

plt.xlabel( 'w' , fontweight= 'bold’ ) 


13 

plt.ylabel( 'Iterations' , fontweight= 'bold' ) 



Executing this code gives Figure 2.1. From figure 2.1 the optimal value of 
parameter u lies some where between 0.8 and 1.1. Hence, zooming more in 
this interval, the optimal value of the parameter uj lies somewhere between 
0.92 and 0.98 as shown in Figure 2.2. 


2.3 The Least Squares Solutions 

If H G R”ixn linear system Ax = b will have a solution a; G R" 

if and only if b is in the column space of matrix A {b G col{A)) [30]. In the 
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W 


FIGURE 2.1: Different values of oj in [0.1,1.5] VS the corresponding numbers 
of iterations. 



0.80 0.85 0.90 0.95 1.00 1.05 1.10 

W 

FIGURE 2.2: Plot of the data (w, Iters) for w in [0.8,1.1] VS the corresponding 
numbers of iterations. 


case that b does not lie in col{A), the problem of solving Ax = b becomes an 
approximation problem, in which we look for some x G R”" such that 

\\b — Ax\\ = min jjlb —Ra:|||, 

3:eR" 
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find a: : 116 —AxII = min {llb —Aa;|||. (2.9) 

There are three interesting cases of 2.9 [47]: 

1. the minimax problem, where the minimization is over the infinity-norm: 

findi : ||6 —Aijjoo = rnin {||6 —Aa;||oo} = rnin { max {\bi — {Ax)i\}}, 

2=1,...,m 


2. the absolute deviation problem, where the minimization is over || • jji: 

m 

find X : ||6-Ai||i = mm ^|bj-(Aa;)j|, 

j=i 


3. the least squares problem, where the minimization is under the classical 
Euclidean distance in R™: 


find X : ||6 —Aa;|l 2 


min 


^£|bj-(Aa;)j|^j 


Given an over-determined linear system: 

Ax = 6, 

where A £ R™^", x G R”, b£ R"^ and m> n, the aim is to find a least squares 
solution of the least squares problem: 

find X : 116 —Aillo = min 
xeR" 

If i G R" is a least squares solution of the linear system Ax = 6, then it 
satisfies: 

||6 —Ai|| < ||6—Axjj, Vx G R" (2.11) 

From the elementary linear algebra, bb = Ax is the orthogonal projection 
of 6 in Col{A) {b—bb = b — Ax _L Col{A)). That is b— Ax is orthogonal to each 
column of A, hence, if aj is the j*^-column of matrix A, then, aj ■ {b— Axb) = 
0, j = 1,... ,n. As a resuit, we get: 

A^{b — Ax) = 0, 


^|6j-(Ax)j'2 

3 = 1 


( 2 . 10 ) 


or 

A^Ax = A^b ( 2 . 12 ) 

Equation (2.12) is called the normal equations, and the least squares solu¬ 
tion X is equi valent to solve the normal equations. 
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Example 2.3 Find the least squares solution of the inconsistent linear system 
Ax = b, where: 


'12 4' 


■ 5 ■ 

3 1 5 


3 

1 1 1 

and b = 

3 

2 2 1 


1 

3 1 3 


4 


Solution: The normal equations are given by Ax = A^b, where, 


■ 24 

13 

31 ■ 


■ 31 ■ 

13 

11 

19 

and A^b = 

22 

31 

19 

52 
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The least squares solution x is obtained by solving the normal equations. In 
MATLAB: 

» A = [1 2 4; 3 1 5; 1 1 1; 2 2 1; 3 1 3] 

A = 

12 4 

3 15 

111 
2 2 1 
3 13 

» b = [5;3;3;1;4] 
b = 

5 

3 

3 
1 

4 

» xh = A’*A\(A’*h) 
xh = 

-0.1639 

0.8970 

0.7507 

In Python, the solution of the least squares problem is obtained by using 
the following Python commands: 

In [9]: import numpy as np 

In [10]: A = np.arrayC[[1, 2, 4], [3, 1, 5], [1, 1, 1], 

[2, 2, 11, [3, 1, 3]]) 

In [11]: b = np. arrayC [ [5] , [3] , [3] , [1] , [4] ] ) 

In [12]: xh = np.linalg.solve(A.T@A, A.TQb) 

In [13]: print(xh) 

[[-0.16388616] 
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[ 0.8969578 ] 

[ 0.75073602]] 

Remark 2.1 The inconsistent linear system Ax = b has a unique least- 
squares solution, if A is a full-rank matrix. That is A is nonsingular 
[30]. 

Remark 2.2 If i is a least-squares solution of the inconsistent linear system 
Ax — b, then ||b—A&II 2 defines the least squares error [53]. 

For example, the least squares error in the above example is obtained by 
using the following MATLAB command: 

>> LSError = norm(b-A*xh, 2) 

LSError = 2.6570 

In Python, the least squares error is obtained by using the command: 

In [14]: LSError = np.linalg.norm(b-A@xh, 2) 

In [15]: print(LSError) 

2.6570401973629143 

2.3.1 Some Applications of Least Squares Solutions 

In this section, Three applications of the least squares Solutions will be dis- 
cussed, namely, fitting linear models to data, polynomials to data and approx- 
imating a function by a polynomial. 

i Fitting a linear model to data: The problem is described as follows: 
Given a set of data points 


Xl 

X 2 


Xn 

yi 

y 2 


Vn 


find a and /3 in R, such that the linear model y = a + i3x gives the best 
fit of the linear model to the data i= l,...,n. This problem is 

equivalent to the problem: 


Find a,P GR: 


j=i 


min 

a,/3€M. 


E 

.j=i 


{yj-{a + /3xj)y 


The parameters a and /3 are called the regression coefficients. To 
find them, we substitute the data points {xj,yj), j = 1... ,n] in the linear 
model y = a + j3x. Then, we get: 

yi = a + Pxi 
2/2 = 0 + PX2 


Vn — 


a + l3xn 


(2.13) 
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Equations (2.13) can be written in the matrix form: 

Ax = y 


where, 


1 Xi 

1 ^2 [q,] 

A= . . , x = \ A and y = 


The regression coefficients a and /3 can be obtained from the normal equa¬ 
tions 

A^ Ax = A^y, 

where, 


A^ A = 


1 1 


Xi X2 X, 


1 Xl 

1] 1 X2 


E n 
1=1 


A^y = 


1 1 ... 1] 2/2 

Xl X2 ... Xn ■ 


E n 

j=i2/i 

E n 

1=1^ jVj 


(A-A) 


det{A'^A) 


adj{A^ A) = 






E n 

j=i^j 


The regression coefhcients a and 0 can be obtained from the solution: 


as follows: 


(A^AfA^y. 


E n 2 ^^n ^^n ^^n 

j=i Xj Ej=i Vj - Ej=i Ej=i xjVj 
«E"=iai2-(Ei=iaii) 

E n -s^n ^^n 

^ ^ j=i xjVj - Ej=i ^3 Ej=i Vj 


(2.16) 
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We write a MATLAB function LinearRegCoef s that receives two vectors 
X and y and returns the regression coefficients ahat and bhat: 


1 function [ahat, bhat] = LinearRegCoefs(x, y) 

2 n = length(x) ; 

3 ahat = ( sum (x.^ 2) *sum (y) -sum (x) *sum (x.*y))/ (n*sum (x."2) 

4 - s um (X)"2) ; 

5 bhat = ... 

(n*sum (x. *y) -sum (x) *sum (y) ) / (n*sum (x. "2) -sum (x) ''2) ; 


Now, we test the function LinearRegCoef s, using data points of heights (in 
meters) and weights (in kilograms) taken for nine students as in Table 2.1. 


TABLE 2.1: Heights (in centimeters) vs. weights (in kilograms) for nine male 
students in secondary school 


Height 

1.65 

1.67 

1.68 

1.72 

1.77 

1.82 

1.86 

1.89 

1.90 

Weight 

57.0 

61.0 

64.0 

69.0 

75.0 

83.0 

90.0 

97.0 

100.0 


1 H = [1.65, 1.67, 1.68, 1.72, 1.77, 1.82, 1.86, 1.89, 1.90] ; 

2 W= [57.0, 61.0, 64.0, 69.0, 75.0, 83.0, 90.0, 97.0, 100.0] ; 

3 [ahat, bhat] = LinearRegCoefs(H, W) ; 

4 WW = ahat + bhat * H ; 

5 plot (H, W, 'mo', H, WW, 'b-', 'LineWidth', 2, ... 

'MarkerFaceColor' , 'm' ) 

6 axis([l,6, 1.95, 50, 110]) 

7 xlabel ( ' Height (in meters)', ' fontweight ' , ' bold ' ) 

8 ylabel ( 'Weight (in kilograms)', 'fontweightbold' ) 

9 grid on 

10 set (gea, 'GridLineStyle' , '— ') ; 

11 set(gea, 'fontweight', 'bold') ; 


In Python, the code to compute the regression coefficients, plotting the 
linear model and scattering the data is as follows: 


1 import numpy as np 

2 import matplotlib.pylab as plt 

3 def LinearRegCoefs(x, y): 

4 n = len (x) 

5 ahat = ... 

(sum (x*x) *sum (y) -sum (x) *sum (x*y) )/ (n*sum (x*x) -sum (x)* *2) 

6 bhat = (n*sum (x*y) -sum (x) *sum (y))/ (n*sum (x*x) -sum (x)* *2) 

7 return ahat, bhat 

8 

9 H = np.array([1.65, 1.67, 1.68, 1.72, 1.77, 1,82, 1.86, ... 

1.89, 1,90]) 

10 W = np.array([57.0, 61.0, 64.0, 69.0, 75.0, 83.0, 90.0, ... 

97.0, 100.0]) 

11 ahat, bhat = LinearRegCoefs(H, W) ; 

12 WW = ahat + bhat * H ; 
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13 plt.plot(H, W, 'mo'/ lw=2) 

14 plt.plot{H, WW, 'b-', lw=2) 

15 plt.axis{[1.6, 1.95, 50, 110]) 

16 plt.xlabel( 'Height (in meters)', f ontweight= 'bold' ) 

17 plt.ylabel( 'Weight (in kilograms) ', fontweight= 'bold' ) 

18 plt.grid(True, ls='--') 


Executing the above Python code, will give the a similar graph as in 
Figure 2.3. 

ii Fitting a polynomial to data: Given a table of data points: 


Xi 

X2 



2/1 

2/2 


Vn 


it is possible to fit a polynomial model with degree not exceeding n — 1 to 
the data. Supposing that the relationship between the variable y and the 
variable x is of the form: 

y = aQ + a\x + a 2 x‘^-\ -l<fc<n —1, (2-17) 

where the regression coefhcients do,...,dfe are to be computed, such 
that model 2.17 is a least-square solution. We substitute the data points 
{xj,yj), j = l,...,n in 2.17 to give n equations in the k+1 unknowns 

yi = ao + aiXi+a2xl-\ - \-akXi 

2/2 = ao + aiX2 + a2X2-l - ha/^x^ 


Vn = aQ + aiXn + a 2 Xn-\ - 'takX^ 



Height (in meters) 


FIGURE 2.3: The linear model vs. the scattered data of heights and weights. 
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Equations (2.18) can be written in the matrix form: 


yi 


T 

Xl 

xj . 

.r.k' 

■ Xl 


ao 

y2 

= 

1 

X2 

xl . 

J-2 


ai 

yn_ 


_i 

Xn 

xl ■ 

.y.k 

Xn. 


ak_ 


(2.18) 


The regression coefficients aQ,...,ak can be found by solving the normal 
equations: 

1 1 ... 

xi x\ ... 


^TL 


■ 1 

1 . 

. 1 ■ 


T 

Xl 

xl . 

. 


ao 

Xl 

xl . 

. x\ 


1 

X2 

xl . 

rrk 

■ ^2 


ai 

_Xn 

xl ■ 

1 


_1 

Xn 

xl ■ 

1_ 


_ak_ 



yi 


2/2 


yn_ 


(2.19) 


Example 2.4 In this example, we write MATLAB and Python codes to 

fit a model y = aQ + aix-\ -hafeX^, for k = 2,4,7 and 8, to random data 

from the normal distribution, as in Table 2.2: 


TABLE 2.2: Virtual data of class centers vs. frequencies 


X 

5 

15 

25 

35 

45 

55 

65 

75 

85 

95 

y 

2 

6 

10 

16 

25 

21 

10 

5 

3 

2 


The MATLAB code is: 


1 ciear ; clc ; clf ; 

2 X = [5 15 25 35 45 55 65 75 85 95] ; 

3 y = [2 6 10 16 25 21 10 5 3 2] ; 

4 XX =1 : 100 ; 

5 deg =[2479] ; 

6 Model = zeros(4, length(x)) ; 

7 Mdl = zeros(4, length(xx)) ; 

8 for 1=1:4 

9 k = deg(1) ; 

10 A = zeros (length (x), k+1) ; 

11 for j = 1 : k+1 

12 A{:, j) = X (: ) . ' (j-1) ; 

13 end 

14 Q = {A'*A)\{A’*y(:)) ;% polyfit{x, y, k) ; 

15 for j = 1 : k+1 

16 Model(l, :) = Model{l, :) + Q {j) . *x . " ( j-1) ; 

17 end 

18 Mdl(l, :) = pchip(x, Model(1, :), xx) ; 

19 FittingError(1) = norm (y-Model(1, :), 2) ; 

20 end 

21 plot (x, y, 'ro', XX, Mdl(l, :), ':b', xx, Mdl(2, :), '— r’, . 

XX, Mdl(3, :), '-.m', xx, Mdl(4, :), '-k', ... 

'MarkerFaceColor' , 'y', 'MarkerSize' , 8, 'LineWidth', ... 

2 ) ; 
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22 legend ( 'Data' , ['k = ' num2str (deg(1))], ['k = ' ... 

num2str (deg (2) ) ] , ['k = ' num2str (deg (3) ) ] , ['k 

num2str (deg(4))], ’fontweight’ , ’bold' ) ; 

23 xlabel('x', 'fontweightbold' ) ; 

24 ylabel('Y', 'fontweightbold’ ) ; 

25 grid on ; 

26 ax = gea ; 

27 ax.FontWeight = ’bold' ; 

28 ax.FontSize = 14 ; 

29 set (gea, 'GridLineStyle' , ' — ') ; 

30 axis([0, 100, 0, 30]) ; 


The above code uses the piece-wise cubic Hermite interpolating polyno- 
mials (pchip) to approximate the solution by a smooth curve, instead of 
the linear interpolation in which the solution curve between two points is 
approximated by a straight line. By exeeuting the above code, we obtain 
the Figure 2.4: 

The Python code to fit polynomials of degrees 2, 4, 7 and 9 to the tabular 
data and plot the least-square curves is: 


1 import numpy as np 

2 from seipy.interpolate import pehip 

3 import matplotlib.pyplot as plt 


4 X = np.array([5., 
75., 85., 

15. , 
95.]: 

25., 

1 

35., 

45 . , 

55 . , 

65 

5 y = np.array([2., 

6., 

10., 

16., 

25., 

21. , 

10 


5., 3., 2.]) 

6 XX = np.arange (1., 101.) 

7 deg = np.array([2, 4, 7, 9]) 

8 Model = np.zeros((4, len(x)), float) ; 



FIGURE 2.4: Fitting polynomials of degrees 2, 4, 7 and 9 to the data in 
Table 2.2. 
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9 Mdl = np.zeros((4, len(xx)), float) ; 

10 for 1 in range(4) : 

11 k = deg [ 1 ] 

12 A = np.zeros( (len (x), k+1), float) ; 

13 for j in range(k+l) : 

14 A[ : , j] = X [ : ] ** {j) ; 

15 Q = np.linalg.solve(A.T@A,A.T@y[:]) ;# polyfit(x, y, k) ; 

16 for j in range(k+l) : 

17 Model[l, :] = Model[l, :] + Q[j]*x**(j) ; 

18 Mdl[l, :] = pchip (x, Model[l, :]) (xx) ; 

19 plt.plot(x, Y, 'ro'/ mfc = 'y\ Iw = 3, label = 'Data') 

20 plt.plot{xx, Mdl[0, : ] ':b', label = 'k = 2', Iw = 2) 

21 plt.plot{xx, Mdl[l, :], ' — r', label = 'k ^ 4', Iw = 2) 

22 plt.plot{xx, Mdl[2, :], '-.m', label = 'k = 1', Iw = 2) 

23 plt.plot{xx, Mdl[3, : ] '-k', label = 'k ^ 9', Iw = 2) 

24 plt. legend () ; 

25 plt.xlabel( ' X ' , fontweight = 'bold') ; 

26 plt.ylabel( ' y ' , fontweight = 'bold') ; 

27 plt.grid(True, Is = '--') 

28 plt.axis{[0, 100, 0, 30]) ; 

29 plt. Show () 


The variable FittingError contains the least-squares errors in fitting the 
polynomials of degrees 2, 4, 7 and 9 to the tabulated data in Table 2.2. 
Those least-squares errors are listed in Table 2.3 

TABLE 2.3: Least squares errors corresponding to fitting with polynomials of 
degrees 2, 4, 7 and 9 


Degree 

2 

4 

7 

9 

LS Error 

1.29467 X 10^ 

6.89477 

3.01662 

1.50627 X 10“^ 


Both MATLAB and Python have a function polyf it that receives two vec- 
tors X = {xi,.. .,Xn),y = (yi, • • -^yn) and a positive integer k and returns 
the optimal parameters di,..., 0 :^+ 1 . In MATLAB, the code lines: 


1 

A = 

zeros ( length (x) , k+1) ; 

2 

for 

j = 1 : k+1 

3 

A{: 

j) = x(:) (j-1) ; 

4 

end 


5 

Q = 

(A'*A)\(A'*y(:) ) ; 

can 

be replaced by: 

1 

Q = 

polyfit (x, y, k) ; 
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In Python, the code lines: 


1 A = np.zeros((len(x), k+1), float) ; 

2 for j in range(k+l) : 

3 A[:, j] =x[:]**(j) ; 

4 Q = np.linalg.solve(A.T@A,A.T@y[:] ) ; 


can be replaced by: 


1 Q = np.polyfit(x, y, k) 


Therefore, we have the following remarks: 

(a) As the degree of the fitting polynomial increases, the least-squares 
error decreases. 

(b) if given n data points, the degree of the fitting polynomial cannot 
exceed n — 1, otherwise, the matrix of the normal equations will be 
singular. 

(c) if given n data points, and the fitting polynomial is of degree n — 1, 
the matrix of coefficients is called the Vandermonde matrix. 

(d) The Vandermonde matrix is nonsingular, but as the number of data 
points n increases, it gets closer to be singular. 

iii Least-squares Approximations of functions: 

Given a function f{x) that is continuous in an interval [a, h\ C R. If Pn{x) = 

-\ -hoo) oq > • ■ •) ^ R is a polynomial of degree n that 

approximates f{x) in [a,b], then the approximation error E(ao,ai,... ,an) 
is defined by: 



dx (2.20) 


The problem is to find the optimal coefficients doidi,... ,dn such that the 
n*^-degree polynomial Pn{x) = ^ least squares solution of 

the approximation problem i.e.\ 



2 


P(.^0y ) • • • 7 


dx 




min {A(Q!o,ai,...,Q;n)} (2-21) 


At the optimal parameters ao,... ,an, we find that 


= 0, fc = 0,... ,n 


(2.22) 
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dE 

dak 




d 

dak 

rb 






2 fix)-Yl 


Q.jX^ 1 


i=o 


dx 


pb pb 

= 2 / x^ f{x)dx— 2''^aj / x^'^^ dx 

Ja j_g Ja 

pb « iJ^k + l _ j+k + 1 

= 2 / x^f{x)dx — 2^^aj - . ^ ^ -=0 (2.23) 


j=0 


j + A: + 1 


From Equation (2.23) we obtain the normal equations: 


u u, 2 

1,2 2 1 3 3 

0 —a b —a 

2 3 



n+1 


n+2 


n+3 




'do' 


Jafix)dx 

n+1 


di 


j!^xf{x)dx 

n+2 


a2 

= 

Ja x^f{x)dx 

^2„ + l_^2„ + l 





2n+l 


^n 


Ja a:"/(a;)dx_ 


(2.24) 


The optimal coefhcients ao,...,an are obtained by solving the normal 
equations (2.24). 

We write a function FunApproxCoef that receives a function f, limits of 
interval a and b and the degree of the least-squares approximating poly- 
nomial n. The function returns a vector alph whose components are the 
optimal coefhcients ao,...,an- 

The MATLAB code of the function FunApproxCoef: 


1 function alph = FunApproxCoef(f, a, b, n) 

2 A = zeros(n+l) ; 

3 y = zeros(n+l, 1) ; 

4 j = 0 ; 

5 while j < n 

6 k = 0 ; 

7 while k < j 

8 A{j + 1, k+1) = {bM j+k+1)-a''{j+k+1) ) / {j+k+1) ; 

9 A(k+1, j+1) = A{j+1, k+1) ; 

10 k = k + 1 ; 

11 end 

12 j = j + 1 ; 


13 


end 
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14 for k = 1 : n+1 

15 y(k) = integral{@(x) x." (k-1) .*f(x) , a, b) ; 

16 end 

17 alp = A\y ; 

18 alph = zeros(l, length(alp)) ; 

19 for j = 1 : length(alp) 

20 alph(j) = alp(n-j+2) ; 

21 end 


The Python code of the function FunApproxCoef: 


1 import numpy as np 

2 from scipy import integrate 

3 import matplotlib.pyplot as plt 

4 

5 def FunApproxCoef(f, a, b, n): 

6 A = np.zeros ( (n+1, n+1), float) 

7 y = np.zeros ( (n+1, 1), float) 

8 j = 0 

9 while j < n: 

10 k = 0 ; 

11 while k < j : 

12 A[j, k] = (b**(j+k+1)-a**(j+k+1))/(j+k+1) 

13 A[k, j] = A[j, k] 

14 k += 1 

15 j += 1 

16 for k in range(n+l) : 

17 y[k] = integrate.quad( lambda x: x**k*f(x), a, b)[0] 

18 alp = np.linalg.solve(A,y) 

19 alph = list (alp) 

20 alph.reverse () 

21 alph = np.array(alph) 

22 return alph 


To test the function FunApproxCoef, we find approximations to the expo- 
nential function f{x) = e^in the interval [0,3], using polynomials of degrees 
1,2 and 4. The MATLAB code to do this is: 


1 a = 0; b = 1 ; 

2 X = linspace(0, 3) ; 

3 f = @ (x) exp (x) ; 

4 P = zeros (4, length(x)) ; 

5 for 1=1:4 

6 n = 1 ; 

7 alph = FunApproxCoef(f, a, b, n) ; 

8 P(l/ :) = polyval (alph, x) ; 

9 end 

10 plot (t, f(t), ' -b ' , t, P(l,:), ’— r', t, P(2, :), '-.m', t, 

P(4, :), ':k', 'LineWidth', 2) ; 

11 xlabel('x', ' fontweight ' , ' bold ’ ) ; 

12 ylabel('y', 'f ontweightbold ' ) ; 

13 legend ( ' y = exp (x)', 'n=l’, 'n=2', 'n=4') ; 

14 grid on ; 
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15 ax = gea ; 

16 ax.FontWeight = 'bold' ; 

17 ax.FontSize = 14 ; 


The Python code is: 


1 import matplotlib.pyplot as plt 

2 a, b = 0, 1 

3 t = np.linspace(0, 3, 301) 

4 f = lambda t: np.exp(t) 

5 P = np.zeros((4, len(t)), float) 

6 for 1 in range(4) : 

7 n = 1+1 

8 alph = FunApproxCoef(f, a, b, n) 

9 P[l/ :] = np.polyval(alph, t) 

10 plt.plot(t, f(t) , ' -b ' , Iw = 2, label= ' y=exp(x) ' ) 

11 plt.plot{t, P[0, :], '—r', Iw = 2, label = 'n = 1') 

12 plt.plot{t, P[l, :], '-.m', Iw = 2, label= ' n= 2') 

13 plt.plot{t, P[3, :], ':k', Iw = 2, label = 'n = 4') 

14 plt.xlabel( ' X ' , fontweight = ’bold') 

15 plt.ylabel( ' y ' , fontweight= 'bold') 

16 plt. legend () 

17 plt.grid(True, Is = '--') 


The exponential function f{x) = is approximated by polynomials of 
degrees 1,2 and 4, and the approximating polynomials are shown in Fig- 
ure 2.5. 


Example 2.5 In this example, we find approximations of the function 
f{x) =cos(7ra:) in the interval [0,1], using polynomials of degrees n = 
1,3,5,...,15. We show the errors ||cos(Tra:) — P„(x) for xG [0,1] and plot 
the approximate functions Pi(x) and Psi^x). 



FIGURE 2.5: Approximation of the exponential function /(x) = with poly¬ 
nomials of degrees 1, 2 and 4. 
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The Python code is: 


1 

a, b = 0 , 1 


2 

t = np.linspace(a, b, 101) 


3 

f = lambda t: np.cos(np.pi*t ) 


4 

P = np.zeros((7, len(t)), float) 


5 

printCn \t | | f (t)-P_n (t) | | _2 ) \n ' ) 


6 

print ( ' - \n ’ ) 


7 

for 1 in range(7) : 


8 

n = 2 *1+1 


9 

alph = FunApproxCoef(f, a, b, n) 


10 

P[l, :] = np.polyval(alph, t) 


11 

print (n, ' \t ' , ... 



'{:7 .4e} ' . format (np.linalg.norm { f(t ) -P2 [ 1, :]) 

'\n')) 

12 

plt.plot(t, f(t), ' -m ' , lw=2, label= ' cos (pi*t ) ' ) 


13 

plt.plot(t, P[0, :], '--r', Iw = 2, label = 'n = 1') 


14 

plt.plot(t, P[2, :], ':b', Iw = 4, label = 'n = 3') 


15 

plt. xlabel ( ' X ' , fontweight = 'bold') 


16 

plt.ylabel ( ' y ' , fontweight= 'bold') 


17 

plt.xticks(np.arange (0 .0, 1.1, 0.1)) 


18 

plt.legend() 


19 

plt.grid(True, Is = '--') 



The resuit of executing the code is: 
n IIf(t)-P_n(t)II_2) 


I 1.4890e+00 

3 4.8910e-02 

5 7.4470e-04 

7 6.5437e-06 

9 3.7348e-08 

II 7.2770e-08 

13 1.8264e-07 

15 3.8105e-08 

The approximate polynomials ^ 1 ( 3 ::) and P 3 {x) are explained in Figure 2.6, 
where Pi (a;) is plotted with a dashed line and ^ 3 ( 0 ;) is plotted with a dotted 
line. The function cos(7ra:) is plotted with a solid line. 

The MATLAB code for approximating f{x) = cos(7ra:) with polynomial of 
degrees n = 1,..., 13 is: 


1 ciear ; clc ; clf ; 

2 a = 0; b = 1 ; 

3 t = linspace(a, b) ; 

4 f = @ (t) cos (pi*t) ; 

5 P = zeros(8, length(t)) ; 

6 1 = 1 ; 

7 fprintf ( 'n \tI I f(t)-P_n(t) II _2\n' ) ; .. 

fprintf ( '-\n ' ) ; 
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FIGURE 2.6: Approximate functions of f{x) = cos(7rx), x S [0,1] with poly- 
nomials of degrees 1 and 3 


8 for k = 1 : 8 

9 n = 2*k-l ; 

10 alph = FunApproxCoef(f, a, b, n) ; 

11 P (k, :) = polyval (alph, t) ; 

12 fprintf { ' %i\t\t%7 . 4e\n ' , n, norm {f{t)-P(k, :), 2)) 

13 end 

14 plot(t, f{t), '-m', t, P{1, :), '--r', t, P(2, :), ':b', 

'LineWidth', 2) ; 

15 xlabel('x', 'f ontweight ' , ' bold ' ) ; 

16 ylabel('Y', ' fontweightbold ’ ) ; 

17 legend('y = cos(pi*x) 'n = 1', 'n = 3') ; 

18 grid on ; 

19 ax = gea ; 

20 ax.FontWeight = 'bold' ; 

21 ax.FontSize = 14 ; 


We notice that the approximate solution improves as we increase the 
degree of the polynomial from n = 1 to n = 9, After that the approxi¬ 
mate solution does not improve. In fact, when we run the MATLAB code, 
we see warning messages connected to n= 11,13 and 15. 

n IIf(t)-P_n(t)II_2 


I 8.7431e-01 

3 2.9178e-02 

5 4.5233e-04 

7 4.0581e-06 

9 2.3696e-08 

II 3.0460e-08 
> In FunApproxCoef (line 17) 

In PlotApproxcos (line 10) 

Warning: Matrix is close to singular or badly scaled. 
Results may be inaccurate. RCOND = 3.190019e-19. 
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13 1.6090e-07 

> In FunApproxCoef (line 17) 

In PlotApproxcos (line 10) 

Warning: Matrix is close to singular or badly scaled. 

Results may be inaccurate. RCOND = 5.742337e-20. 

15 3.1356e-07 

The reason behind these error messages will be discussed in the next chap- 
ter. 


Remark 2.3 When approximating a function /(x) by a polynomial Pn{x) 
of degree n in the interval [0,1], the resulting coefficient matrix is called 
Hilbert matrix. The Hilbert matrix of type n x n (77«) is given by: 


77« = 


1 


3 
1 

4 


1 

! 

4 
1 

5 


1 -| 

n 

1 

n^l 


(2.25) 


Ln n+1 n-\-2 


2n+l-l 


MATLAB has a function hilb, which recieves an integer n and return the 
corresponding Hiblbert matrix 77«. 


» hilb(3) 


ans = 


1.0000 

0.5000 

0.3333 



0.5000 

0.3333 

0.2500 



0.3333 

0.2500 

0.2000 



» hilb(5) 





ans = 





1.0000 

0.5000 

0.3333 

0.2500 

0.2000 

0.5000 

0.3333 

0.2500 

0.2000 

0.1667 

0.3333 

0.2500 

0.2000 

0.1667 

0.1429 

0.2500 

0.2000 

0.1667 

0.1429 

0.1250 

0.2000 

0.1667 

0.1429 

0.1250 

0.1111 
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Ill-Conditioning and Regularization 
Techniques in Solutions of Linear Systems 


Abstract 

If a small perturbatiori is introduced to either the coefficient matrix A or 
vector b, it might lead to a big change in the solution vector x. Hence, both 
the direct and iterative methods are not guaranteed to give accurate solution 
of the given linear system. 

This chapter is divided into two sections. The first section presents the 
concept of ill-conditioning in linear Systems and how to use MATLAB® and 
Python to measure the condition numbers of matrices. In the second section, 
some regularization techniques are presented to stabilize the Solutions of ill- 
conditioned Systems. 


3.1 Ill-Conditioning in Solutions of Linear Systems 

An ill-conditioned linear system is a linear system which responds to a small 
perturbation on the coefhcient matrix or the vector at the right-hand side with 
a large change in the system solution [28, 42]. To see this, two kinds of little 
perturbations will be presented to two examples that will be considered. In 
the first example a slight change on one component of the coefficient matrix 
will be introduced and in the second example the perturbation will be made 
on the vector at the right-hand side. 


Example 3.1 In this example we consider the linear system Fx = d, with 


5.0 

1.0 

1.0\ 

/ 5.0 

1.0 

5.0 

1.0 and b = 

-3.0 

1.0 

1.0 

5.0/ 

l 5.0 


To solve this linear system in MATLAB, we use the following commands: 


» A = [5. , 1., 1.; 1., 5., 1.; 1. , 1. , 5.] 
A = 
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5 11 

15 1 

115 

» b = [5.; -3.; 5.] 
b = 

5 

-3 

5 

>> X = A\b 
X = 

1.0000 

-1.0000 

1.0000 

In Python: 

In [1]: import numpy as np 

In [2]: A = np.arrayC[[5., 1., 1.], [1., 5., 1.], [1., 1., 5.]]) 
In [3]: b = np.arrayC[[5.], [-3.], [5.]]) 

In [4]: X = np.Iinalg.soIveCA, b) 

In [5]: print(’x = \n’, x) 

X = 

[[ 1 .] 

[-1.] 

[ 1 .]] 

Now, the first perturbation we consider will be on the component An, 
where we set B = A and Bn = An + 10“^ = All + 0.0001 

» B = A 
B = 

5 11 

15 1 

115 

» B(l, 1) = B(l, 1) + 0.0001 
B = 

5.0001 1.0000 1.0000 

1.0000 5.0000 1.0000 

1.0000 1.0000 5.0000 

>> y = B\b 

y = 

1.0000 
-1.0000 
1.0000 

>> disp(’x - y = ’), disp(num2str(x-y)) 
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X - y = 

2.1428e-05 

-3.5714e-06 

-3.5714e-06 

>> disp([’|l x-y||_2 = ’ num2str(norm(x-y, 2 ))]) 

II x-yl|_2 = 2.2015e-05 

In Python, 

In [7]; B = A.copyO 

In [8]: B[0, 0] += le-4 

In [9]; y = np.Iinalg.solve(B, b) 

In [10]; print(’y = \n’, y) 

y = 

[[ 0.99997857] 

[-0.99999643] 

[ 1.00000357]] 

In [11]: print(’x - y = \n’, x-y) 

X - y = 

[[ 2.14281123e-05] 

[-3.57135204e-06] 

[-3.57135204e-06]] 

In [12]; print(’ II x-y II _2 = \n’, np.linalg.norm(x-y, 2)) 

I Ix-yl L2 = 

2.201529253996403e-05 

The second perturbation under consideration will be on the right-hand 
side, where 63 will be replaced by 63 + 10 “^ giving a vector c in the right- 
hand side. The original coefficient matrix A will remain without a change and 
the linear system Az = c will be solved. 

>> c = b ; 

» c(3) = c(3) + le-4 
c = 

5.0000 

-3.0000 

5.0001 


» z = A\c 

z = 

1.0000 
- 1.0000 
1.0000 
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» disp(’x - z = ’), disp(num2str(x-z)) 

X - z = 

3.5714e-06 

3.5714e-06 

-2.1429e-05 

>> disp([’|l x-z||_2 = ’ num2str(norm(x-z, 2))]) 
II x-zl|_2 = 2.2016e-05 


In Python, 

In [12]; c = b.copyO 

In [13]; c[-l] += le-4 

In [14]; z = np.linalg.solve(A, c) 

In [15]; print(’z = \n’, z) 
z = 

[[ 0.99999643] 

[-1.00000357] 

[ 1.00002143]] 

In [16]; print(’x - z = \n’, x-z) 

X - z = 

[[ 3.57142857e-06] 

[ 3.57142857e-06] 

[-2.14285714e-05]] 

In [17]; print(’IIx-zII_2 = \n’, np.linalg.norm(x-z, 2)) 
IIx-zl|_2 = 

2.2015764296112095e-05 


From Example 3.1 it can be noticed that small changes in some components 
of the coefficient matrix or the vector at the right-hand side lead to small 
changes in the solution. Hence, the given linear system is not sensitive to 
small perturbations. A system that is not sensitive to small perturbations is 
called well-posed system. 


Example 3.2 In this example a linear system Fx = d is considered, where 


F = 


1001 

-999 

999 \ 

/1001 

1 

1 

—1 and d= 

1 

1000 

-1000 

1000/ 

\iooo 


The purpose from this example is to show how a small change in one 
entry of the coefficient matrix F or vector d can cause a drastic change in the 
solution of the linear system Fx = d. 

MATLAB is used to solve the linear system Ax = b, with the commands; 

» F = [1001, -999, 999; 1, 1, -1; 1000, -1000, 1000] ; 

» d = [1001; 1; 1000] ; 

>> X = F\d 
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Warning: Matrix is close to singular or badly scaled. Results 
may be inaccurate. 

RCOND = 1.067985e-16. 


X = 

1 

0.4147 

0.4147 

The warning indicates that the matrix is close to being singular, hence the 
results might be inaccurate. The MATLAB solution of the linear system is far 
from the exact solution [1.0,1.0,1.0]”^ as noticed. 

In Python, the above linear system can be solved by using the Python 
commands: 

In [1]: import numpy as np 
In [2]: A = np.array([ 

[l.OOle+03, -9.990e+02, 9.990e+02], 

[l.OOOe+00, l.OOOe+00, -l.OOOe+00], 

[l.OOOe+03, -l.OOOe+03, l.OOOe+03]]) 

In [3]; b = np.arrayC[[1001], [1], [1000]]) 

In [4]; X = np.Iinalg.soIveCA, b) 

In [5]; print(’x =\n’, x) 

X = 

[[ 1 . ] 

[0.93969727] 

[0.93969727]] 

Again, despite Python having better accuracy in finding a solution of the 
linear system than MATLAB, stili the error in solving the linear system is not 
small. 

Again, small perturbations will be introduced to the coefficient matrix 
F and vector d. First, a perturbation 10“® will be added to Fu giving a 
matrix G. Then, computing the solution of the linear system Gy — d and 

Wy-xh- 

» G = F ; 

>> format long g 
» G(l, 1) = G(l, 1) + le-5 
G = 

1001.00001 -999 999 

0.9999999999999 1.0000000000001 -0.9999999999999 

1000 -1000 1000 
>> y = G\d 

Warning: Matrix is close to singular or badly scaled. Results 
may be inaccurate. 

RCOND = 1.092847e-16. 
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y = 

0.999997714598233 
22853063.4012923 
22853063.4012946 
>> disp(norm(y-x, 2)) 

32319111.6174026 

In Python, the commands will be used: 

In [6]; G = F.copyO 

In [7]; G[0, 0] += le-5 

In [8]; y = np.Iinalg.soIv(G, d) 

In [9]; printC’y = ’, y) 

y = 

[[9.99997654e-01] 

[2.34012849e+07] 

[2.34012849e+07]] 

In [10]: print(’y-x = \n’, y-x) 
y-x = 

[[-2.34606334e-06] 

[ 2.34012840e+07] 

[ 2.34012840e+07]] 

In [11]: print("7o20.16e"7o np.Iinalg.norm(x-y, 2)) 

3.3094413214227043e+07 

Second, a small value 10“® is added to the third component of vector d to 
get a vector g and solve the linear system Fz = g. In MATLAB, the following 
commands are used: 

» g = d ; 

» g(3) = g(3) + le-5 

g = 

1001 

1 

1000.00001 
» z = F\g 

Warning: Matrix is close to singular or badly scaled. Results 
may be inaccurate. 

RCOND = 1.067985e-16. 

z = 

0.999997659959457 
23385189.5233867 
23385189.523389 
>> format long e 

>> dispCMIz - x||_2 = ’), disp(norm(z-x, 2)) 

I|z - x||_2 = 

3.307165159616157e+07 
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In Python: 

In [12] : g = d. copyO 

In [131: g[-l] += le-5 

In [14]: z = np.linalg.solve(F, g) 

In [15]: print(’z = \n’, z) 
z = 

[[ 9.99977257e-01] 

[-2.94647268e+ll] 

[-2.94647268e+ll]] 

In [16]: printC’I Ix-zl |_2 = \n’, "7.20.16e"7. 

np.linalg.norm(x-z, 2)) 

IIx-zl|_2 = 

4.1669416392824341e+ll 

Python does not release a message to complain about the closeness of 
the matrix to being singular, but it is true that the matrix is close to being 
singular. 

The results obtained in Example 3.2 show that either a small change in 
the coefficient matrix or the vector at the right-hand side lead to huge changes 
in the solution of the linear system. Sensitivity to small changes indicate that 
such a linear system is ill-conditioned [42]. 

3.1.1 More Examples of Ill-Posed System 

In this section two least squares approximation problems are considered. In the 
first problem, a Vandermonde matrix will be constructed to find the regression 
coefhcients that give best fit of tabular data. In the second example, a Hilbert 
matrix will be constructed to find the coefficient of a polynomial of specific 
degree to approximate a given function. 

Example 3.3 In this example, the data given in Table 2.1 is considered. 
It consists of nine data pairs of heights and weights for nine persons. The 
purpose is to find regression coefhcients og ,...,oq such that W — 
is a least squares solution. In MATLAB we compute the regression coefhcients 
as follows: 

» H = [1.65, 1.67, 1.68, 1.72, 1.77, 1.82, 1.86, 1.89, 1.9]’ ; 

» W = [57, 61, 64, 69, 75, 83, 90, 97, 100]’ ; 

>> V = vander(H) ; 

» A = V’*V ; 

» b = V’*W ; 

>> X = A\b 

Warning: Matrix is close to singular or badly scaled. Results 
may be inaccurate. 

RCOND = 1.040749e-20. 
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X = 

3918.1 

-21520 

35375 

-46361 

2.1855e+05 

-4.3689e+05 

52491 

5.8691e+05 
-4.1416e+05 

In Python: 

In [17]: import numpy as np 

In [18]: H = np.array([1.65, 1.67, 1.68, 1.72, 1.77, 1.82, 1.86, 
1.89, 1.9]) 

In [19]; W = np.arrayC[57, 61, 64, 69, 75, 83, 90, 97, 100]) 

In [20]; V = np.fIipIr(np.vander(H)) 

In [21]; A = V.TQV 

In [22]; b = V.TQW 

In [23]; x = np.linalg.solve(A, b) 

In [24]; print(’x = \n’, x) 

X = 

[-1.26559701e+06 3.39786878e+06 -3.76621053e+06 2.29662532e+06 

-8.88505759e+05 2.23976721e+05 -1.87100975e+04 -9.10775041e+03 

2.24200821e+03] 

Now we present a small change on the third component of the height vector 
and see how this change will affect the resulting regression coefficients. 

» HI = H ; 

» Hl(3) = Hl(3) + 0.01 ; 

>> VI = vander(Hl) ; 

» B = V1’*V1 ; 

» bl = V1’*W ; 

» y = B\bl 

Warning; Matrix is close to singular or badly scaled. Results 
may be inaccurate. RCOND = 1.439393e-20. 

y = 

2214 

-13856 

28739 

-31202 

90976 

-2.4644e+05 

2.4675e+05 

-20840 

-63193 
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>> disp(norm(x-y)) 
7.6365e+05 


In Python: 


In [25] : HI = H.copyO 
In [26]: HI[2] += 0.01 
In [27]: VI = np.fIipIr(np.vander(Hl)) 

In [28]: B = Vl.TOVl 
In [29]: bl = Vl.TSW 
In [30]: y = np.linalg.solve(B, bl) 

In [31]: printCy = \n’, y) 

y = 

[-1.67279152e+05 2.06762159e+05 1.09086145e+05 -2.55301791e+05 

8.12931633e+04 3.93681517e+04 -2.63131683e+04 3.30116939e+03 

2.38312295e+02] 

In [32]: print(’||x - yll_2 = ’{0:1.6e}format(np.linalg 

.norm(x-y))) 

IIx - yl L2 = 5.821901e+06 

Example 3.4 In this example, we approximate the function y{x) = 5xe~^^ 
by a polynomial P(x) of degree 12 in an interval The coefhcients of the 

polynomial are computed by the function FunApproxCoef developed in the 
previous chapter. 

The coefhcient matrix A is of type 13 x 13 defined by 


Aij — 




-,i,j = l...,13 


i + j-l 

and the right-hand side is a vector b of type 13 x I defined by 


bj= x^ ^y{x)dx,j = 

J a. 

At the beginning the problem is solved without any change in either matrix 
A or vector b. Then we make a small change in the last component in b such 
that bi 3 = 6 i 3 + 10 “^. 

The MATLAB code to compute the polynomials is: 


1 f = @ (x) 2 + 5 * X.* exp(-2 * x.''2) ; 

2 A = zeros(13, 13) ; b = zeros{13, 1) ; 

3 a=0.0;b=2.0; 

4 al = 0.0 ; bt = 2.0 ; 

5 for i = 1 : 13 

6 for j = 1 : i 

7 k = i + j- l; 

8 A(i, j) = (bfk - al''k)/k ; 

9 A(j, i) = A(i, j) ; 

10 end 

11 b(i) = integral(@(x) x.^(i-l) .*f (x), al, bt) ; 

12 end 
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13 C = A\b(:) ; C = wrev(C) ; 

14 bl = b ; bl (end) = bl (end) + le-4 ; 

15 C1 = A\bl (:) ; C1 = wrev(Cl) ; 

16 t = linspace{al, bt, 401) ; F = f(t) ; 

17 Pe = polyval{C, t) ; Pp = polyval(Cl, t) ; 

18 

19 subplot(l, 2f 1) ; plot (t, F, '— b', 'LineWidth', 3) ; 

20 xlabel ( ' x ' , ' fontweight ' , ' bold ' ) ; ylabel ( ' y ’ , ... 

' fontweight ' , ' bold ' ) ; 

21 legend(’y(x) = 5xe''{-2x''2}' ) ; grid on ; ax = gea ; 

22 ax.FontWeight = ’bold' ; ax.FontSize = 12 ; 

23 set {gea, 'XTiek', linspaee(al, bt, 9)) ; set (gea, 'YTick', ... 

linspaee (1, 5, 9)) ; 

24 axis([al, bt, 1, 5]) ; 

25 

26 subplot(l, 2, 2) ; plot (t. Pe, ' — r', t, Pp, ':k', ... 

' LineWidth ' , 3) ; 

27 xlabel('x', ' fontweightbold ' ) ; ylabel{'y', ... 

' fontweightbold ' ) ; 

28 legend ( ’ Without perturbation ' , ’With perturbation ' ) ; grid on ; 

29 ax = gea ; ax.FontWeight = 'bold' ; ax.FontSize = 12 ; 

30 set (gea, 'XTiek', linspaee (al, bt, 9)) ; set (gea, 'YTiek', ... 

linspaee (1, 5, 9)) ; 

31 axis{[al, bt, 1, 5]) ; 


The Python code is: 


1 import numpy as np 

2 import matplotlib.pyplot as plt 

3 from seipy import integrate 

4 f = lambda x: 2.+5*x*np.exp (-2*x**2) 

5 A, b = np,zeros{(13, 13), 'float'), np.zeros{(13, 1), 'float') 

6 al, bt = 0., 2. 

7 for i in range(13) : 

8 for j in range(i+l) : 

9 k = i+j+1 

10 A[i, j] = (bt**k-al**k)/k 

11 A[ j, i] = A[i, j] 

12 b[i] = integrate.quad( lambda x: x**i*f(x), al, bt)[0] 

13 C = np.linalg.solve(A, b) 

14 e = list (C.T [0] ) 

15 e. reverse () 

16 C = np.array(e) 

17 t = np.linspaee(al, bt+(bt-al)/400, 401) 

18 F = f (t) 

19 Pe = np .polyval (C, t) 

20 b [-1] += le-4 

21 C1 = np.linalg.solve(A, b) 

22 e = list (C1 .T [0] ) 

23 e. reverse () 

24 C1 = np.array(e) 

25 Pp = np.polyval(C1, t) 

26 

27 plt.figure(1, figsize=(20, 8)) 

28 plt.subplot(1, 2, 1) 

29 plt.plot(t, F, '— b', Iw = 3, label = 'y{x) = 5 x e''{-2x''2}' ) 

30 plt.xlabel( 'x' , fontweight= ' bold ' ) 
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31 plt.ylabel ( ' y ' , fontweight= ' bold ’ ) 

32 plt. legend () 

33 plt.xticks(np.arange(al, bt+(bt-al)/8, (bt-al)/8), ... 

fontweight= ' bold ' ) 

34 plt.yticks(np.arange(1., 5., 0.5), fontweight= ' bold ' ) 

35 plt.grid(True, ls=':') 

36 plt.axis([al, bt, 1., 5.]) 

37 

38 plt. subplot (1, 2, 2) 

39 plt.plot(t. Pe, '— b', Iw = 3, label = 'Without perturbation ' ) 

40 plt.plot(t, Pp, ':k', Iw = 3, label = 'With perturbation') 

41 plt.xlabel( ' X ' , fontweight= ' bold ' ) 

42 plt.ylabel( ' y ' , fontweight= ' bold ' ) 

43 plt.legend( loc="upper center") 

44 plt.xticks(np.arange(al, bt+(bt-al)/8, {bt-al)/8), ... 

fontweight= ' bold ' ) 

45 plt.yticks(np.arange(1., 5., 0.5), fontweight= ’ bold ' ) 

46 plt.grid(True, ls=':') 

47 plt.axis([al, bt, 1., 5.]) 

Figure 3.1 contains two subgraphs. In the first subgraph (at the left side) the 
original function y{x) = 5xe~‘^^ is plotted. In the second subgraph (at the 
right side) the coefRcients resulting from the linear Systems without and with 
changing the right-hand side are used to graph the approximating polynomials. 


3.1.2 Condition Numbers and Ill-Conditioned Matrices 

In this section we give an interpretation as to why some linear Systems are 
more sensitive to small changes than other linear Systems. Before we do so, 
we discuss the following concepts: 

(I) Matrix norm: If 4. is a matrix of dimensions m x n, with elements 
ai^j,i = 1,...,to; j = 1,...,n, then: 

(a) ||4|ii = max {YT=i\a-islYT=i |ah2|,• ■ -.YJILi W^,n\} 




FIGURE 3.1: Left: the exact function. Right: the approximating polynomials 
of degree 12 without and with changes in the last component of vector b. 
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(c) II2III2 = max| where aj is the eigenvalue of 

A-A^. 

We can use the Python command norm to compute ||A|jp of matrix A. 

In [31]: from numpy.linalg import norm 

In [32]: A = np. array( [ [1, -2], [2, -3], [3, -4]]) 

In [33]: nl, n2, n3 = norm(A, 1), norm(A, 2), norm(A, np.inf) 

In [34]: print(nl, ’\n’, n2, ’\n’, n3) 

9.0 

6.54675563644 

7.0 

In MATLAB, also a function norm is used to compute ||A|jp of matrix 

A. 

» A = [1, -2; 2, -3; 3, -4] ; 

>> nl = norm(A, 1), n2 = norm(A, 2), n3 = norm(A, inf) 
nl = 

9 

n2 = 

6.5468 

n3 = 

7 

Let A and B be two matrices of type mx n, a he scaler and a; be a 
vector in R. The matrix norm is a function |j • || : R that satisfies 

the following properties: 


(a) 

IV 

0 


(b) 

II 

|a|-||A||. 

(c) 

\\A+B\\ 

T 

VI 

(d) 

IA 

Pllll^li- 

(e) 

VI 

l!^lll|s|l- 


(II) Machine precession: A digital computer has limited storage capac- 
ity to represent a number with floating point. Subject to the Inter¬ 
national Electrical and Electronic Engineering (IEEE) standards, a 
double-precession number is represented by 8 bytes (64 bits). These 64 
bits are divided into three segments: the most significant bit (the most 
left bit) is used for representing the sign of the number. The next II 
bits are used to represent the exponent part and the last 52 bits (called 
the mantissa) are used to represent the binary fraction bits. Under 
this Setup, the minimum floating number that can be represented by 64 
bits is approximately —1.7977 x and the largest floating number 
is approximately 1.7977 x 10®°®. 
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Both MATLAB and Python adopt the IEEE double precision numbers 
as default type of their numerical variables with floating points. The 
smallest binary fraction that can be represented by 52 bits is 2“®^ « 
2.2204 X 10“^®. This number is called the machine precision and is 
denoted by s. In MATLAB the machine precision can be seen by typing 
eps in the command window: 

>> disp(eps) 

2.2204e-16 

In Python it can be seen by typing: 

In [35]: print(np.finfo(float).eps) 

2.220446049250313e-16 

If the fractional part of some floating number requires more than 52 bits 
to be represented, then the computer can represent only 52 bits of the 
number and ignores the rest. The difference between the true fractional 
part of the number and the computer representation is called the round 

off error. For example it is well known that sinnTr = 0,n = 0,±1,_ 

But Python and MATLAB do not give 0 exactly if we choose ni^O: 

In [36] np.sin(O.O) 

Out[36]: 0.0 

In [37]: np.sin(np.pi) 

Out[37]; 1.2246467991473532e-16 

>> disp(sin(0.0)) 

0 

>> disp(sin(2*pi)) 

-2.4493e-16 

Now to understand how a small change in either vector b at the right-hand 
side of the linear system Ax = b, or in matrix A, can change the solution of 
the linear system, the properties of matrix norm are used to explore such a 
phenomena. 

First, it is assumed that a change in the right-hand side from b to b + Ab 
can lead to a change in the solution of the system from x to x-|-Ax. Then, 

A(x-|-Ax) = b + Ab^ Ax-I-AAx = b+ Ab^ AAx = Ab^ Ax = A~^Ab 

because Ax = b. Then, 

||b|| = ||Ax||<||A|!||x|K||A||||x||>|ib||, (3.1) 

||Ax|| = ||A-iA6||<|!A-i|||!Ab| 


and 


(3.2) 
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By dividing Equation (3.2) by Equation (3.1) gives: 

ll^li ^ \\A-^-\\Ab\\ 
ll^llll^ll - \\b\\ 

from which, 

i)^<MIII|A-||M (3,3) 

The number k{A) = ||yl||||^“^|| is called the condition number of matrix 
A. The bigger the condition number, the higher sensitivity of the linear system 
to perturbation on the right-hand side and vice versa. Equation (3.3) shows 
the relationship between the relative change in solution of the linear system 
compared to the relative change in the right-hand side of the linear system. 

Second, if it is assumed that matrix A is changed to A -|- AA leads to 
change in the solution from a: to a; -|- Aa;, then 

{A + AA){x + Ax) = b^ Ax + AAx + AA{x + Ax) 

= AAx + AA{x + Ax) = 0 

Multiplying the two sides by A~^ gives 

Aa;-|-A~^A^(a;-|-Ax) = 0 —Ax = ^~^AA(x 4-Ax) 


By taking the norms for the two sides, we get: 

||Ax|| = p-iAA(x +Ax)|j < ||A-i|l • ||A4l|| • Ijx-f Ax|| 


Dividing the two sides by ||x-}-Ax|| and multiplying the right-hand side by 


A 
A ■ 


gives: 


l|Ax|| 
||x-|-Ax| 


< k(A) 


l|AA| 

PII 


(3.4) 


Equation (3.4) shows the relationship between the relative change in the 
solution vector x compared to the relative change in matrix A, in terms of 
the condition number of matrix A. 

It can be seen that the smaller the condition number of a matrix A, the less 
the linear system will be sensitive to perturbations, and the resulting system is 
well-posed. Also, the larger the condition number, the more sensitive the linear 
system is to perturbations, and the resulting system is ill-posed. The condition 
number of the 3x3 zero matrix is oo and for the 3x3 identity matrix is 1. 
The MATLAB command cond can be used for finding the condition number 
of a matrix A. 


» A = [100 1 0; 0 1 2; 0 2 4.003] 
A = 

100.0000 1.0000 0 

0 1.0000 2.0000 
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0 2.0000 4.0030 

>> cond(A) 
ans = 

1.6676e+05 
>> condCzeros(3)) 
ans = 

Inf 

>> cond(eye(3)) 
ans = 

1 


The Python command numpy. linalg. cond can be used for computing the 
condition number of a matrix A. 

In [38]; A = np.arrayC[[100, 1, 0], [0, 1, 2], [0, 2, 4.003]]) 

In [39]; print(’{0;1.6e}’.format(np.linalg.cond(A))) 

1.667617e+05 

In [40]; print(’{0;1.6e}’.format(np.linalg.cond 
(np.zeros((3, 3))))) 

inf 

In [41]; print(’{0;1.6e}’.format(np.linalg.condCnp.eye(3)))) 
l.OOOOOOe+00 

A matrix with a large condition number is called ill-conditioned, otherwise, 
it is well-conditioned [7, 6] . The condition number for any singular matrix is 
oo. For the identity matrix, it is 1. For any nonsingular matrix, the condition 
number lies in [l,oo). 

An equivalent measure to the condition number k{A) is the reciprocal 
condition number (rcond), where 

™"‘'('4>=PiirpRir 

Under this measure, the reciprocal condition number of any singular matrix 
is 0, for the identity matrix is 1 and for any other nonsingular matrix is in the 
interval (0.0,1.0). A matrix A is closer to singular if 0 k(A) 0 [42]. 

3.1.3 Linking the Condition Nnmbers to Matrix Related 
Eigenvalues 

If A is a squared n by n matrix with eigenvalues {Ai,..., A^}, then the condi¬ 
tion number of matrix A is the ratio between the largest (in magnitude) and 
smallest (in magnitude) eigenvalues. That is; 

k{A) = - -, 

'^min 

where Ama® = max{|Aj|,j = 1,... ,n} and Amm = min{|Aj |, j = 1,... ,n}. 
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If is an m by n matrix, where mi^n, and {Ai,..., Xm} are the eigenvalues 
of A ■ A'^ , then the condition number of matrix A is given by: 


k{A) 



where Xmax = max{|Aj|, j = and Xmin = min{Aj,j = 1,.. .,m} 

In Example 2.5 the function f{x) = costtx is approximated with polynomi- 
als Pn{x)-,n=l,3,..., 15. It was expected that as the degree of the polynomial 
increases, the accuracy of the solution increases. But, what was noticed was 
that the accuracy of the approximating polynomial increases up to degree 9, 
after that the accuracy drops. That happened because of the increase in the 
condition number as the polynomial degree increases. 

The following MATLAB commands show the condition numbers of the 
Hilbert matrices Hn for n = 2,..., 13 


>> fprintf(’n\t\t I|H_n||\n’); for n = 2 ; 13 

fprintf (’7,i\t\tyolO. 6e\n’, n, cond(hilb(n) ) ) ; end 
n llH_n|| 


2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 


1.928147e+01 

5.240568e+02 

1.551374e+04 

4.766073e+05 

1.495106e+07 

4.753674e+08 

1.525758e+10 

4.931541e+ll 

1.602520e+13 

5.227499e+14 

1.629550e+16 

1.682118e+18 

2.715269e+17 

2.777738e+17 


In Python, the code is: 


In [41]: from numpy.linalg import cond 
In [42]: from scipy.Iinalg import hilbert as hilb 
In [43]: for n in range(2, 16): printCn, ’\t\t’, 
’{0:1.6e}’.format(cond(hilb(n)))) 


2 

3 

4 

5 

6 

7 

8 
9 


1.928147e+01 

5.240568e+02 

1.551374e+04 

4.766073e+05 

1.495106e+07 

4.753674e+08 

1.525758e+10 

4.931534e+ll 
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10 

1.602503e+13 

11 

5.220207e+14 

12 

1.621164e+16 

13 

4.786392e+17 

14 

2.551499e+17 

15 

2.495952e+17 


This shows that as the dimensions of Hilbert matrices increase, they become 
more ill-conditioned. It is also noticed that MATLAB and Python do agree 
on the values of condition numbers as long as rcond{Hn) > eps. When 
rcond{Hn) < eps they could have different roundoff errors, causing them to 
produce different condition numbers. 

Another example is the Vandermonde matrix, used with least-squares 
approximations. As the number of data points increases, the condition num- 
ber of the corresponding Vandermone matrix increases, so it becomes more 
ill-conditioned. The following MATLAB commands show the condition num¬ 
bers of Vandermone matrix for different numbers of data points: 

>> fprintf(’n\t\t I|V_n||\n’); for n = 2 : 13 

fprintf (’7oi\t\tyolO. 6e\n’, n, cond(vander(H(l :n) ) ) ) ; end 
n IIV_n|| 


2 

3.755673e+02 

3 

1.627586e+05 

4 

3.051107e+07 

5 

3.481581e+09 

6 

2.480023e+19 

7 

2.970484e+21 

8 

6.929557e+24 

9 

7.174378e+27 

10 

8.795704e+31 

11 

2.868767e+35 

12 

1.380512e+39 

13 

1.532255e+42 


In Python: 


In [44]: for n in range(2, len(H)): printCn, 
’{0:1.6e}’.format(cond(vander(H[:n])))) 


2 

3 

4 

5 

6 

7 

8 

9 

10 
11 


3.755673e+02 

1.627586e+05 

3.051107e+07 

3.481581e+09 

2.479949e+19 

2.970484e+21 

6.929557e+24 

7.174382e+27 

8.796918e+31 

2.862293e+35 


’\t\t’ , 
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12 1.375228e+39 

13 1.740841e+42 

If the condition number of some matrix A is of order 10^, then when 
solving a linear system Ax = 6 up to t' decimal places from the right can be 
inaccurate (with the notice that 16 decimal places are truly represented for 
double-precision numbers). To see this, we consider a Vandermonde matrix 
generated by a random vector v: 

>> V = randClO, 1) 

V = 

4.4559e-01 

6.4631e-01 

7.0936e-01 

7.5469e-01 

2.7603e-01 

6.7970e-01 

6.5510e-01 

1.6261e-01 

1.1900e-01 

4.9836e-01 

>> V = fliplr(vander(v)) ; 

In Python, the vector v and the Vandermonde matrix V of v can be gen¬ 
erated by using the Python commands: 

In [45]: v = np.random.rand(10) 

In [46] : v 
Out [46] : 

arrayC[0.47585697, 0.31429675, 0.73920316, 0.45044728, 0.16221156, 
0.8241245, 0.9038605, 0.28001448, 0.85937663, 0.07834397]) 

In [47]: V = np.fliplr(np.vander(v)) 

We can measure the condition number of matrix V in MATLAB: 

>> Cv = cond(V) 

Cv = 

1.730811304916736e+10 
In Python: 

In [48]: cV = np.linalg.cond(V) 

In [49]: print(’{0:1.6e}’.format(cV)) 

8.671331e+07 

Let X be a column vector of ones of dimension 10 and b = Vx. 

» X = onesClO, 1) ; 

» b = V * x ; 
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Each component of 5 is a sum of row elements of V. 

Now let us pretend that we don’t know x, and we want to retrieve it by 
solving the linear system: 

Vy = b 

The vector y is given by: 
y = V\b ; 

To find out how close y to x, we measure the infinity norm of the difference 
between them. 

>> Error = norm(x-y, inf) 

Error = 

6.900445270741074e-07 

which teli that the Error is of 0(10“^). That means the accuracy has been 
lost in 9 decimal places. 

The above steps can be executed in Python, by using the commands: 

In [50]: X = np.ones((10, 1)) 

In [51] : b = V @ X 

In [52]: y = np.linalg.solve(V, b) 

In [53]: Error = np.linalg.norm(y-x, np.inf) 

In [54]: print(’{0:1.6e}format(Error)) 

3.949043e-09 

3.1.4 Further Analysis on Ill-Posed Systems 

Given a linear system of equations 

Fx = y (3.5) 

where F € a; G R” is a vector of unknowns, y G R™ is a vector of exact 

data and n<m. 

We want to compute a solution x* G R" to the linear sytem (3.5) such 
that, 

(1) X* is a least-squares solution to problem (3.5). The existence of a least- 
squares solution x^s is characterized by 

X* = xls = sxg roin WFx — yW"^ (3.6) 

xi^S is s. least squares solution. 

(2) X* is unique. The uniqueness of a least-squares solution xlsmn is char¬ 
acterized by 

X* = xlsmn =s.rg miii {||a;is||2} = F“^y (3.7) 

xlsmn is referred to as a least squares minimum norm solution. 
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(3) the computed solutiori x* is stable. This is characterized by the existence 

of F-\ 

Problem (3.5) is a well-posed problem if conditions (l)-(3) are satisfied. It 
is an ill-posed problem if it is not a well-posed problem. 

In real life, the data y is usually obtained from measurements which 
are contaminated by small errors [42]. Solving the inverse problem using 
the noisy data in an ill-posed System will resuit in a catastrophic erroneous 
solution which is irrelevant to the true solution, as had seen in the previous 
section. 

To find an interpretation to what is going behind, one should consider the 
singular value decomposition of the matrix F. Let (LI,S, be the singular 
value decomposition of the matrix F, where U G and V G R”^" are 

two unitary matrices and S G Rmxn jg diagonal matrix. The inverse of the 
matrix F is given by F~^ = (UTiV^)~^ = . Now the solution x of 

the linear system Fx — y is given by 


X = 




{ujy) 

CTi 


Vi 


(3.8) 


where Uj and Vi are the columns of the matrices U and V, and the aiS 
are the singular values of F, i= 1,..., m, with ui > cr2 > ■ ■ ■ > CTn > 0. 

Therefore, the solution a; is a linear combination of {V-i,...,Vn}, the 

U^y 

columns of V, with coefhcients ^ = 1,... ,n. 

When the noisy data y'^ is used instead of the exact data y, we get 




*=i ti 

(3.9) 


From equation (3.9), we find that, ||y —ailH = the 


matrix F tends to be singular, some of its singular values tend to be zeros, 

and hence, some of the coefficients gets very large. This telis us that, 

the residual norm is not effected by only how small the noise is, but also by 
how small a singular value of the matrix F is. 

As an explanation, let us look at the singular values of the 20 x 20 Hilbert 
matrix: 


» H = hilb(20) ; 

» [U, S, V] = svd(H) ; 
>> format short e ; 

>> D = diag(S) 

D = 

1.9071e+00 
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4.8704e-01 

7.5596e-02 

8.9611e-03 

8.6767e-04 

7.0334e-05 

4.8305e-06 

2.8277e-07 

1.4140e-08 

6.0361e-10 

2.1929e-ll 

6.7408e-13 

1.7384e-14 

3.7318e-16 

1.5057e-17 

1.2511e-17 

7.3767e-18 

5.4371e-18 

1.7279e-18 

5.8796e-19 

We see that the last 6 singular values of H{= USV'^) are below e(= 2.2204 x 
10“^®), which are not discriminated from 0 in the 64 —Systems. Dividing 
by such a singular value when computing H~^ = VS~^U'^ causes huge errors, 
since the diagonal elements of S~ ^ are the reciprocals of the diagonal elements 
oiS. 

According to the distribution of the singular values of F, ill-posed 
problems (whose coefficient matrices are ill-conditioned) are divided into 
two classes. Namely, the rank deficient problems and the discrete ill 
posed problems. These two classes of problems can be distinguished by using 
the following properties [36] 

1. in the rank deficient problems, there is a small cluster of small singular 
values, while in discrete ill-posed problems there is a large cluster of small 
singular values. 

2. in the rank deficient problems there is a ciear gap between the large and 
small singular values, while in discrete ill-posed problems the singular 
values decay gradually without gaps between the large and small singular 
values. 

3. for the rank deficient problems, there could be a formulation which can 
eliminate the ill-conditioning but no such formulation exists for the ill- 
posed problems. 
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3.2 Regularization of Solutions in Linear Systems 

Regularization is the process of stabilizing the solution of an ill-posed prob- 
lem [60, 59, 58] 

Fx = y,F and yGR”". 

The regularization is done through introducing a parametric family of approx¬ 
imate inverse operators 

{r„ : R*" ^ R™ : r„ « F-^,a G (0,oo)} 

such that, for each satisfying jjy'^ — yjj < e there exists a G (0,oo) such that, 
a;“ '^= Tay"^ —> a; as e —>■ 0. 

Some regularization techniques include the truncated SVD method, the 
Tikhonov regularization method, the L-curve method and the Morosov dis- 
crepancy principle [8, 9]. 

In this section we discuss these methods. 


3.2.1 The Truncated SVD (TSVD) Method 

Considering the linear system 

Fx = b, Fg a; g R*" and y g R*", 


and supposing that C/, S and V are the svd factors of matrix F, such that 
F = C/ST^. Hence, F~^ = VT,~^U'^. The unique solution of the linear system 
fx = b is given by: 


x = VY.-^U'^b = J2 

j=i 


u 


^b 


u 


Tb 


0-1 


-Vi¬ 


ti 


^b 


where Uj and Vj are the columns of U and V respectively, and aj is the 
diagonal element of S, with the notice that (T\> a 2 > ■ ■ ■> (Jn- 
The idea behind the regularization method based on the truncated SVD is 
to truncate terms that contain very small singular values from the summation 
in equation (3.9). That is if a > £ (e is the machine precision) and cti > (T 2 > 
■ ■■>(7£>a> cr^+i > • •. > cTn, then 


X 


l 


E 



uTb ujb 

-vi H-1— —v^ 

CTi ae 


(3.10) 


Up to some optimal regularization level a*, the TSVD method performs 
well (for a < a*), after that (when a exceeds a*) the solution is affected by 
truncating more terms and as a > a* increases, x°‘ goes away from x. 
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The MATLAB function SolveWithTSVD receives a matrix F, vector b and 
regularization parameter a. It applies the truncated SVD regularization to 
return a solution x. 


1 

function x = SolveWithTSVD(F, b. Alpha) 

2 

[U, 

S, V] = svd(A) ; 

3 

D = 

diag(S) ; 

4 

[m, 

n] = size (A) ; 

5 

X = 

zeros (m, 1) ; 

6 

for 

i = 1 : length ( S ) 

7 


if D ( i) > Alpha 

8 


x = x + U(:, i)'*b/D(i)*V(:, i) ; 

9 

end 


10 

end 



The function SolveWithTSVD is tested for 20 x 20 Hilbert matrix, the vector 
b at the RHS components are the summations of FI rows. The exact solution 
y is a vector of ones of type 20 x 1. 

Using a script SolveForHilb.m an unregularized solution z and a regular- 
ized solution w at level a = e are computed and shown in Figure 3.2. Also, the 
error norms for different values of the regularization parameters aj = 10-^ 
are shown. 


1 H = hilb (20) ; 

2 y = ones (20, 1) ; 

3 b = H * y ; 

4 z = H\b ; 

5 subplot (1, 2, 1) ; 

6 plot{l:20, y, '-b', 1:20, z, 'r:', ' LineWidth ' , 3) ; 

7 legend (’ Exact Solution', 'Unregularized Solution') ; 

8 xlabel (' Solution Component') ; 

9 ylabel (’ Component Value ' ) ; 

10 grid on ; 

11 format short e 

12 disp ( ' Error norm of unregularized solution | |y-z | | _2 ' ) , . . . 

disp (norm (y-z)) 



Solution Component 



1 • • • • RegUarESil Solutlcir 1 

'v 

i 


Solution Component 


FIGURE 3.2: Unregularized and regularized Solutions of Hx = b. 
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13 w = SolveWithTSVD(H, b, eps) ; 

14 subplot(l, 2, 2) ; 

15 plot{l:20, y, '-b', 1:20, w, 'r:', ' LineWidth ' , 3) ; 

16 legend ( ' Exact Solution', 'Regularized Solution') ; 

17 xlabel (’ Solution Component') ; 

18 ylabel (’ Component Value ' ) ; 

19 grid on 

20 disp ( ' Error norm of regularized solution ||y-w | | _2 ' ) , . . . 

disp (norm (y-w)) 

21 

22 X = zeros(20, 15) ; 

23 Err = zeros(15, 2) ; 

24 disp ( ’ Alpha | | x-y | | _2 ' ) ; 

25 disp ( '-' ) ; 

26 for n = 1 : 15 

27 Alpha = 10" (n-1) *eps ; 

28 x{:, n) = SolveWithTSVD(H, b. Alpha) ; 

29 Err(n, 1) = Alpha ; 

30 Err(n, 2) = norm(y-x(:, n) , inf) ; 

31 end 

32 disp (Err) ; 


The script SolveForHilb is executed by using the command: 

>> SolveForHilb 
> In SolveForHilb (line 4) 

Warning; Matrix is close to singular or badly scaled. Results 
may be inaccurate. RCOND = 5.231543e-20. 

Error norm of unregularized solution Ily-z||_2 
4.4218e+02 

Error norm of regularized solution Ily-w|I_2 
3.0301e-01 


Alpha IIx-yII_2 


2.2204e-16 

2.2204e-15 

2.2204e-14 

2.2204e-13 

2.2204e-12 

2.2204e-ll 

2.2204e-10 

2.2204e-09 

2.2204e-08 

2.2204e-07 

2.2204e-06 

2.2204e-05 

2.2204e-04 


1.4427e-01 

2.6146e-02 

3.3896e-04 

3.3896e-04 

2.4002e-06 

7.8511e-06 

7.8511e-06 

4.5607e-05 

2.1812e-04 

2.1812e-04 

1.0205e-03 

4.2420e-03 

1.5665e-02 
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2.2204e-03 5.1211e-02 

2.2204e-02 1.6937e-01 

The Python code is: 


1 import numpy as np 

2 from scipy.linalg import hilbert as hilb 

3 from numpy.linalg import norm, svd 

4 import matplotlib.pyplot as plt 

5 def SolveWithTSVD{A, b, Alpha): 

6 U, S, V = svd(A) 

7 X = np.zeros-like(b) 

8 n = len (S) 

9 for j in range(n) : 

10 if S[j] > Alpha: 

11 X += np . dot (U [ : , j ] , b)/S [ j ] *V, T [ : , j ] 

12 else: 

13 continue 

14 return x 

15 

16 H = hilb (20) 

17 y = np . ones ( (20, ) , 'float') 

18 b = H@y 

19 z = np.linalg.solve(H, b) 

20 Eps = np.spacing(1.0) 

21 Alpha=Eps 

22 w = SolveWithTSVD(H, b. Alpha) 

23 print( 'Error norm for Unregularized solution = norm(y-z)) 

24 print( 'Error norm for Regularized solution = norm(y-w)) 

25 plt. f igure (1) 

26 plt.subplot (1, 2, 1) 

27 t = np.arange(l, len(b)+l) 

28 plt.plot(t/ Y, '-b', lw=3, label=’Exact solution') 

29 plt.plot(t/ z, '-.r', lw=3, label= ' Unregularized solution') 

30 plt.xlabel( ' Solution component ' , fontweight= 'bold ' ) 

31 plt.ylabel( ' Component value ' , fontweight= ' bold ' ) 

32 plt.grid(True, ls=':') 

33 plt. legend () 

34 

35 plt.subplot (1, 2, 2) 

36 t = np.arange(l, len(b)+l) 

37 plt.plot(t, y, '-b', lw=3, label='Exact solution') 

38 plt.plot(t/ w, ':m', lw=3, label= ' Regularized solution') 

39 plt.xlabel( ' Solution component ' , fontweight= 'bold ' ) 

40 plt.ylabel( ' Component value ' , fontweight= ' bold ' ) 

41 plt.grid(True, ls=':') 

42 

43 Err = np.zeros((16, 2), 'float') 

44 for j in range(16) : 

45 Alpha = 10**j*np.spacing(1.) 

46 w = SolveWithTSVD(H, b. Alpha) 

47 Err[j, 0] = Alpha 

48 Err[j, 1] = norm(y-w) 

49 print (Err) 
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Executing the code gives: 

runfile(’D:/PyFiles/regWithTSVD.py’, wdir=’D;/PyFiles’) 

Error norm for Unregularized solutiori = 136.97028071675066 

Error norm for Regularized solution = 0.5943628935591088 

[[2.22044605e-16 5.94362894e-01] 

[2.22044605e-15 4.08083331e-04] 

[2.22044605e-14 4.08083331e-04] 

[2.22044605e-13 4.08083331e-04] 

[2.22044605e-12 1.19446030e-05] 

[2.22044605e-ll 2.17705662e-05] 

[2.22044605e-10 2.17705662e-05] 

[2.22044605e-09 1.12283185e-04] 

[2.22044605e-08 5.33801614e-04] 

[2.22044605e-07 5.33801614e-04] 

[2.22044605e-06 2.34255106e-03] 

[2.22044605e-05 9.49123719e-03] 

[2.22044605e-04 3.54621361e-02] 

[2.22044605e-03 1.21845461e-01] 

[2.22044605e-02 3.83219676e-01] 

[2.22044605e-01 1.09386370e+00]] 

3.2.2 Tikhonov Regularizaton Method 

Tikhonov regularization methods are the most well known methods for the reg¬ 
ularization of an ill-conditioned System. In a Tikhonov regularization method 
a penalty is added to a solution with a large norm. If ip(x) = ||i^x —y|j| is the 
square of the length of the residual vector Fx — y, the problem is to minimize 
the functional (pa = Pa{x) = ||Fa: — y|||-F Q;||£c|j|, where the penalty a||a;|j 2 
is added to penalize the components of the vector x on the rapid variations 
from positive to negative values and vice-versa. 

We write 

(pa{x) =< Fx — y, Fx — y> +a < x,x >= (Fx — y)"^{Fx — y) + ax^^x 

= x^ {f'^ F + al)x — x"^ F^^y — y^ Fx — y^y 
If the functional ipa{x) takes its minimum value at ai = a;“, then 
^^^5) =0 = 2{F'^F + aI)x°‘-2F^y and {F'^F + al)x'^ = F^y. 

X = X^ 

Now, F'^F + aI =V{T.^ + aI)V^, then 

n 

= (3.11) 

We consider the term ■ For large eigenvalues cTi’s, the term ^^'‘^2 ~ ^7 
and for very small values for cr,-, Suitable choices for a do reduce 

' l 

the high frequencies resulted by the small eigenvalues. 
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A python code to solve the linear system Hx = b, with H the Hilbert 
matrix of type 20 x 20 and b the 20 x 1 vector whose components are the sums 
of H rows. 


1 import numpy as np 

2 from scipy.linalg import hilbert as hilb 

3 from numpy.linalg import norm, svd 

4 import matplotlib.pyplot as plt 

5 def SolveWithTikhonov(A, b. Alpha): 

6 U, S, V = svd{A) 

7 X = np. zeros-like (b) 

8 n = len (S) 

9 for j in range(n) : 

10 X += np.dot(U[:,j], b) *S [ j]/(S[j]**2+Alpha)*V.T[: , j] 

11 return x 

12 

13 H = hilb (20) 

14 y = np . ones ( (20, ) , 'float') 

15 b = H@y 

16 z = np.linalg.solve(H, b) 

17 Eps = np . spacing (1.0) 

18 Alpha=Eps 

19 w = SolveWithTikhonov(H, b. Alpha) 

20 print(’ Error norm for Unregularized solution = ', ... 

' {0:1.8e}' . format (norm(y-z) ) ) 

21 print(' Error norm for Regularized solution = ', ... 

'{0:1.8e}' . format (norm(y-w))) 

22 plt. f igure (1) 

23 plt.subplot (1, 2, 1) 

24 t = np.arange(l, len(b)+l) 

25 plt.plot(t/ y, ' -b ' , marker='s’, lw=3, label='Exact solution’) 

26 plt.plot(t/ z, '-.r', marker='o', lw=3/ label= ' Unregularized ... 

solution ' ) 

27 plt.xlabel ( ' Solution component ' , fontweight= ' bold ' ) 

28 plt.ylabel( ' Component value ' , fontweight= ' bold ' ) 

29 plt.grid(True, ls=':') 

30 plt. legend () 

31 

32 plt. subplot (1, 2, 2) 

33 t = np.arange(l, len(b)+l) 

34 plt.plot(t, y, '-.b', marker=’s', lw=3, label='Exact solution’) 

35 plt.plot(t, w, ':m', marker=’o', lw=3, label= ' Regularized ... 

solution ' ) 

36 plt.xlabel ( ’ Solution component ' , fontweight= 'bold ’ ) 

37 plt.ylabel (’ Component value’, fontweight= ’ bold ' ) 

38 plt.grid(True, ls=':') 

39 plt. legend () 

40 

41 Err = np.zeros((16, 2), 'float') 

42 for j in range(16) : 

43 Alpha = 10**j*np.spacing(1.) 

44 w = SolveWithTikhonov(H, b. Alpha) 

45 Err[j, 0] = Alpha 

46 Err[j, 1] = norm(y-w) 

47 print (' (0: 1 .6e} ’. format (Err[j, 0]), '\t', ... 

' (0 : 1 .8e} ’ . format (Err[j, 1])) 
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FIGURE 3.3: Unregularized solution and regularized solution with reg. param 
a = e oi Hx = b. 


In Figure 3.3 the unregularized solution and the regularized solution a;“ 
for a = e are shown. 

The norm error of regularized Solutions a;" where aj = = 1,..., 16 

are computed through the above Python code, and the outputs are as follows: 


Error norm for Unregularized solution 
Error norm for Regularized solution = 


2.220446e-16 

2.220446e-15 

2.220446e-14 

2.220446e-13 

2.220446e-12 

2.220446e-ll 

2.220446e-10 

2.220446e-09 

2.220446e-08 

2.220446e-07 

2.220446e-06 

2.220446e-05 

2.220446e-04 

2.220446e-03 

2.220446e-02 

2.220446e-01 


2.96674549e-04 

4.95587019e-04 

7.25227169e-04 

1.76193763e-03 

2.40230468e-03 

5.05832063e-03 

8.76991238e-03 

1.41598120e-02 

2.96964698e-02 

4.37394775e-02 

9.44807071e-02 

1.42053519e-01 

2.95683135e-01 

4.69592530e-01 

9.24797653e-01 

1.61864335e+00 


= 1.36970281e+02 

2.96674549e-04 


The MATLAB code is: 


1 H = hilb (20) ; 

2 y = ones (20, 1) ; 

3 b = H * y ; 

4 z = H\b ; 

5 subplot (1, 2, 1) ; 

6 plot{l:20, Y;. ' -b ' , 1:20, z, '-.m', ' LineWidth ' , 3) ; 

7 legend ( ' Exact Solution', 'Unregularized Solution') ; 
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8 xlabel ( ' Solution Component') ; 

9 ylabel (’ Component Value ' ) ; 

10 grid on ; 

11 format short e 

12 disp ( ' Error norm of unregularized solution | |y-z | | _2 ' ) , . . . 

disp (norm (y-z)) 

13 w = SolveWithTikhonov(H, b, eps) ; 

14 subplot(l, 2, 2) ; 

15 plot{l:20, y, '-b', 1:20, w, 'r:', 'LineWidth', 3) ; 

16 legend ( ' Exact Solution', 'Regularized Solution') ; 

17 xlabel (’ Solution Component') ; 

18 ylabel (' Component Value') ; 

19 grid on 

20 disp ( ' Error norm of regularized solution ||y-w | | _2 ' ) , ... 

disp (norm (y-w)) 

21 

22 X = zeros(20, 15) ; 

23 Err = zeros(15, 2) ; 

24 disp ( ' - ' ) ; 

25 disp ( ' Alpha | | x-y | | _2 ' ) ; 

26 disp ( '-' ) ; 

27 for n = 1 : 15 

28 Alpha = 10"(n-1) *eps ; 

29 x(:, n) = SolveWithTikhonov(H, b. Alpha) ; 

30 Err(n, 1) = Alpha ; 

31 Err(n, 2) = norm(y-x(:, n) , inf) ; 

32 end 

33 disp (Err) ; 

34 

35 function x = SolveWithTikhonov(A, b. Alpha) 

36 [U, S, V] = svd(A) ; 

37 D = diag(S) ; 

38 [m, n] = size (A) ; 

39 X = zeros (m, 1) ; 

40 for i = 1 : n 

41 x = x + U(:, i)'*b*D(i)/((D(i)) "2+Alpha) *V ( : , i) ; 

42 end 

43 end 


In Tikhonov regularization further smoothing can be imposed on the vector 
of solution X, by imposing the smoothness of its derivative, using the k^^- 
order differential operator, fc = 1,2,.... Then we consider a functional of the 
form 

^a{x) = \\Fx-y\\l + a\\Lx\\l (3.12) 

where L = Dj. is the order differential operator. 

For fc = 1, L = Z?! is the (n — 1) x n differential operator defined by 


Di 


-1 1 0 

0 -1 1 


0 0 
0 0 


_ 0 0 0 ... -1 1 _ 

and for fc = 2, L = D 2 is the (n — 2) x n differential operator defined by 









86 


Ill-Conditioning and Regularization Techniques 


D2 


1-210 
0 1-21 


0 0 0 
0 0 0 


0 0 0 0 ... 1 -2 1 


Using the first-order differential operator L = Di, equation (3.12) takes 
the form 


m—1 


(faix) = \\Fx-y\\l + a\\Dix\\l = \\Fx-y\\l + a ^ {xi+i-Xif (3.13) 


i=l 


Using the second-order differential operator L = D 2 , equation (3.12) takes 
the form 


m —2 


ipa{x) = \\Fx-y\\l + a\\D 2 x \\2 = \\Fx-y \\2 + a ^ {x^+2-‘2^x^+l-x^f 


2 = 1 


(3.14) 

Minimizing ipa{x) which appears in equation (3.12) is equivalent to solving 
the normal equations 


(F^F+ aL^L)a;“ = 


(3.15) 


Hansen [22] considered the use of the generalized singular value decomposi- 
tion (GSVD) to obtain the solution of problem (3.15). The generalized singular 
value decomposition for a pair {F,L), where F £ R."ixn ^ ^ with 

p <n < m is given by the form 


F=u( ^ ^ L = V{M 0)X-i (3.16) 

\ U ^n—p J 

where 



/ 

CTi 

0 . 

. 0 \ 


1 

Mi 

0 . 

. 0 

\ 



0 

(72 . 

. 0 



0 

M2 ■ 

. 0 


s = 





M = 







V 

0 

0 . 

• / 


[ 

0 

0 . 

Pp 

/ 


with 0 < (Ti < (72 < ... < (Tp < 1 and 0 < pp < Pp-i < • •. < yi < 1 
The sets {ai} and {pi} are normalized such that ai+ y,i = 1. 
The solution x°‘ of equation (3.15) is given by 


x'^ = j2(uTy)- 


2 = 1 


■afli 


-X,:+ ^ {Ufy)X, 

i=p+l 


(3.17) 
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Using small values for the regularization parameter a on equations (3.11) and 
(3.15) may not inhibit the quantities and effectively. Hence, not 

much improvement can be obtain for the smoothness of the unregularized solu- 
tion, but it causes the regularized solution to stay close to the unregularized 
solution. On the other hand, using large values for the regularization param¬ 
eter a do impose more smoothness on the regularized solution, but causes it 
to go away from the minimum residual value, obtained for the unregularized 
solution. Between these two extremes, optimal regularization parameters do 
lay. 

When we plot the points (Hfa;" — y||, ||La;“|j) for different values of a on 
the loglog scale, it takes the shape of an L-curve. The L-curve does clearly 
display the compromise between minimization of the two quantities \\Fx°‘ — 
y\\ and ||La:“||) [22]. At the corner of this L-curve, optimal values for the 
regularization parameter a are located. 

3.2.4 The Discrepancy Principle 

Linz and Wang [33] considered the problem of solving a linear system Fx = b, 
where F is an m x n matrix with m > n, a: G R” and b G R™. He stated that, 
a solution x for this linear system is referred to as acceptable if it satisfies the 
condition 

\\Fx — b\\ <e 

where e is the total error of the data and computations. 

According to Linz and Wang [33] a; is a plausible solution only if 

||La;l| <M 

where M is a chosen positive number. Therefore, the task becomes to select 
a, so that the regularized solution a;“ is both acceptable and plausible. 

The solution x is the most plausible acceptable solution, if it solves the 


problem 


minimize = ||La;|| 
cceR" 

(3.18) 

subject to the constraint 


w 

VI 

io" 

1 

(3.19) 

Problem (3.18)-(3.19) is equivalent to solve the problem 


{F'^F + aB'^B)xa = F^^b 

(3.20) 

subject to 

\\Fxa-b\\ =e 

(3.21) 

and the Morozov discrepancy principle is the problem of selecting 
(3.20-3.21) is satisfied. 

a such that. 
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Solving a System of Nonlinear Equations 


Abstract 

This chapter discusses the Solutions of nonlinear Systems using MATLAB® 
and Python. It is divided into two sections. The first section presents four 
numerical methods for solving single nonlinear equation. The second section 
discusses numerical methods for solving a system of nonlinear equations. 


4.1 Solving a Single Nonlinear Equation 

Consider a nonlinear equation of the form: 

f(x) = 0 

where / : R —>■ R is a nonlinear function. The problem is to find a point x = x* 
such that f{x*) = 0. Then, x = x* is called a root of the function /. For 
example, if f{x) = —cosx + O.S, then the root of f{x) = — cosx + O.S = 0 

lies at the intersection points of the curves + 0.5 and cos (x) as can be seen 
in Figure 4.1. 

In this section four methods for solving the given nonlinear equation, 
namely, the bisection method, the Newton-Raphson method, the secant and the 
iterative method will be discussed and implemented in MATLAB and Python. 

4.1.1 The Bisection Method 

Assume that the function f{x) is continuous in an interval [a, b] and is changing 
its sign from positive to negative or vice versa as it moves from a to b. A 
direct resuit from the intermediate value theorem is that a continuous function 
cannot change its sign in an interval, without passing through zero [52]. 

The bisection method is an iterative method, which in each iteration divides 
the interval that contains the root into two equally subintervals, drops the half 
which does not contain the root and looks for the root in the other half. 

If the interval [a, 6] is divided into two equal subintervals [a,c] and [c, &], 
where c= {a + b)/2 is the midpoint of the interval [a, 5]. The root of the 
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FIGURE 4.1: Graphs of the functions e““ + 0.5 (dashed curve) and cos(a:) 
(dash-dotted curve). The x-coordinate of circle at the intersection of the two 
curves is the root of — cosx + O.S = 0 in [4,6]. 


function / either lies in [a,c] or [c,6]. If it lies in [a,c], then /(o) •/(c) < 0. 
In this case, we know that the root does not lie in the interval [c,b] and we 
look for the root in [a,c]. If it lies in [c, 6], then /(c) ■ f{b) < 0 and in this 
case, we know that the root does not lie in [a,c], so we look for it in [c, &]. 
The bisection method continues dividing the interval, which contains the root 
into two sub-intervals, such that one sub-interval contains the root whereas 
the other does not; therefore, the method considers only the interval which 
contains the root and drops the other half. 

The MATLAB code that implements the bisection method is as follows: 


1 function x = Bisection(f, a, b, Epsilon) 

2 while b-a > Epsilon 

3 c = (a+b)/2 ; 

4 if f(a)*f(c) < 0 

5 b = c ; 

6 elseif f(b)*f(c) < 0 

7 a = c ; 

8 else 

9 X = C ; 

10 end 

11 end 

12 X = C ; 

Now, we can call the function Bisection from the command prompt: 

>> format long 
>> Epsilon = le-8 ; 

>> f = @(x) x'2 - 3 ; 

>> r = BisectionCf, 1, 2, Epsilon) 
r = 

1.732050813734531 

>> s = BisectionCf, -2, -1, Epsilon) 
s = 

-1.732050813734531 
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The Python code to implement the Bisection function is: 


1 

def Bisection(f, a, 

b, Eps): 

2 

from math import 

fabs 

3 

c = (a+b)/2.0 


4 

Iters = 1 


5 

while fabs(f (c)) 

> Eps : 

6 

if f(a)*f(c) 

< 0: 

7 

b = c 


8 

else : 


9 

a = c 


10 

c = (a+b)/2 


11 

Iters += 1 


12 

return c, Iters 



In [1]: f = lambda x: x**2 - 3 
In [2]: a, b = 1. , 2. 

In [3]: Eps = le-8 

In [4]: X, Iters = Bisection(f, a, b, Eps) 

In [5]: printC'Approximate root is:’, x, ’\nIterations:’, Iters) 
Approximate root is: 1.7320508062839508 
Iterations: 25 

4.1.2 The Newton-Raphson Method 

If a function f{x) is continuous in the neighbourhood of a point xq, then it 
can be written in the form: 

/(x) = f(xo) + f'(xoXx-xo) + 0((x-xof) 

Now, if xi is a root for the function f(x), then f(xi) = 0. That is: 

f(xo) + f'(xo)(xi-Xo) «/(xi) =0 

From the above equation: 


xi =Xo- 


f(xo) 

f'(xo) 


provided that /'(xq) ^ 0. 

The Newton-Raphson method is an iterative method for finding an approx- 
imation to the closest root of the function /(x) to an initial guess xq- Starting 
from Xq, it generates a sequence of numbers xo,Xi,X 2 ,..where, 




/(Xn) 

/'(Xn) 


This sequence of numbers converges to the closest root of f to xq. 
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To find a root for the function f{x) = x'^ — 3, using the Newton-Raphson 
method, it is noticed that f'{x) = 2x, and therefore, the iterative method is 
of the form: 


,(n+l) _ ,(.) _ _ ,(») _ -3) 

* 2i(«) 

The MATLAB function NewtonRaphson.m, implements the Newton- 
Raphson method: 


1 function [x. Iter] = NewtonRaphson(f, fp, xO, Epsilon) 

2 Iter = 0 ; 

3 xl = xO - f(xO)/fp(xO) ; 

4 while abs(f(xl)) > Epsilon 

5 xO = xl ; 

6 xl = xO - f(xO)/fp{xO) ; 

7 Iter=Iter+l; 

8 end 

9 X = X1 ; 

Calling the NewtonRaphson function from the command prompt: 

>> format long 

>> f = (§(x) x'2 - 3 ; 

» fp = @(x) 2*x ; 

>> Epsilon = le-8 ; 

» xO = 1 ; 

>>[x, Iterations] = NewtonRaphson(f, fp, xO, Epsilon) 

X = 

1.732050810014728 
Iterations = 

3 

>> [x, Iterations] = NewtonRaphson(f, fp, -xO, Epsilon) 

X = 

-1.732050810014728 
Iterations = 

3 


The code of the Python function NewtonRaphson is: 


1 def NewtonRaphson(f, fp, xO, Eps): 

2 from math import fabs 

3 x = xO - f{xO)/fp(xO) 

4 Iters = 1 

5 while fabs(f(x)) > Eps: 

6 xO = X 

7 X = xO - f(xO)/fp(xO) 

8 Iters += 1 

9 return x, Iters 
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Running the above code with xg = 1-0 one time and xq = —1.0 another time: 


In [6]: X, Iters = NewtonRaphson(f, fp, xO, Eps) 

In [7]: printC'Approximate root is:’, x, ’\nIterations:’, Iters) 
Approximate root is: 1.7320508100147276 
Iterations: 4 

In [8]: X, Iters = NewtonRaphson(f, fp, -xO, Eps) 

In [9]: printC'Approximate root is:’, x, 'Xniterations:’, Iters) 
Approximate root is: -1.7320508100147276 
Iterations: 4 

4.1.3 The Secant Method 

The secant method has a close form as the Newton-Raphson method, but it 
does not require the analytical form of the derivative of f(x) at Xn (f'(xn))- 
It replaces f'{xn) by the finite difference formula: 

N fixn)- fixn-i) 

J \'^n) ~ 

Xn—\ 

Hence, the secant method is of the form: 

^n— 1 n/ \ 

Starting from some interval [a,6] that contains a root for /(x), the secant 
method iteration approaches the zero of f{x) in [a, 5]. 

The MATLAB function Secant implements the secant method. It receives 
a function f, the limits of the interval [a, 6] that contains the root of / and a 
tolerance £ > 0. It applies the secant method to return an approximate solution 
X and the number of iterations Iterations 


1 function [x, Iterations] = Secant(f, a, b, Eps) 

2 X = b - ((b-a)*f(b))/(f(b)-f(a)) ; 

3 Iterations = 1 ; 

4 while fabs(f(x)) > Eps 

5 a = b ; 

6 b = X ; 

7 x=b- {(b-a)(b))/{f(b)-f(a)) ; 

8 Iterations = Iterations + 1 ; 

9 end 


Calling the MATLAB function to find the approximate root of — 3 = 0 as 
follows. 

>> f = @(x) x'2-3 ; 

» a = 1; b = 2 ; 

>> Eps = le-8 ; 

>> [x, Iterations] = Secant(f, a, b, Eps) 
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X = 

1.732050807565499 
Iterations = 

5 


The Python code of the function Secant is as follows. 


1 def Secant (f, a, b, Eps) : 

2 from math import fabs 

3 X = b - {{b-a)*f(b))/{f{b)-f(a)) 

4 Iterations = 1 

5 while fabs(f(x)) > Eps: 

6 a, b = b, X 

7 x=b- {(b-a)(b))/{f(b)-f(a)) 

8 Iterations += 1 

9 return x, Iterations 

To find the root of the equation — 3 = 0, the following Python instructions 
are used: 

In [10]: f = lambda x: x**2-3 
In [11] : a, b = 1. , 2. 

In [12]: Eps = le-8 

In [13]: X, Iterations = Secant(f, a, b, Eps) 

In [14]: print('Approximate root is:’, x, ’ \nIterations :’, Iterations) 
Approximate root is: 1.732050807565499 
Iterations: 5 

4.1.4 The Iterative Method Towards a Fixed Point 

A point X* is a fixed point for a function g{x), if 

g{x*)=x* 

Suppose that the function f{x) is differentiable and can be written as 

f{x) = g{x)-x 


If x\ is a root of f{x), then, 

f{xi) =g{xi)-xi = 0 ^X 1 =g{xi) 

That means xi is a fixed point for the function g{x). The idea behind the 
iterative method towards a fixed point is to write the function f{x) in the 
shape g{x) — x, and starting from some initial guess xq, the method generates 
a sequence of numbers xo,xi,X 2 , ■ ■ ■ that converges to the fixed point of the 
function g{x), using the iterative rule: 


Xn-\-l — giXn) 
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To show how the the iterative method works, it will be applied to find the 
roots of the function 

f{x) = x'^-3 

first the function — 3 is written in the form g{x) —x. One possible choice is 
to write: 


f{x) = x^ — 3 = a;^ + 2a: + l — 2a; — 4=(a: + l)^ — 2a; — 4 
If X* is a root for f{x), then: 


f{x*) = {x* + lf-2x*-4 = 0 


from which 

2 

Starting from an initial point xq, the iterative method to find a root for f{x) 
is: 

(n+i) _ +1)^-4 

“ 2 

The following MATLAB code, computes a root of the function f{x) = x^ — 3 
using the iterative method: 


1 

function [x, Iterations] = IterativeMethod(g, xO, 

Epsilon) 

2 

Iterations = 0 ; 


3 

X = g(xO) ; 


4 

while abs(x-xO) > Epsilon 


5 

xO = X ; 


6 

X = g(xO) ; 


7 

Iterations = Iterations + 1 ; 


8 

end 


From the command prompt, the following instructions are 

typed: 

>> 

g = (§(x)((x+l)'2-4)/2 ; 


>> 

xO = 1 ; 


>> 

Epsilon = le-8 ; 


>> 

[x, Iterations] = IterativeMethodCg, xO, Epsilon) 


-1 

732050811416889 



Iterations = 
58 


In Python, the code of the function IterativeMethod is: 


1 def IterativeMethod(g, xO, Eps): 

2 from math import fabs 

3 X = g (xO) 

4 Iters = 1 
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5 while fabs(x - xO) > Eps: 

6 xO = X 

7 X = g(xO) 

8 Iters += 1 

9 return x, Iters 


Running the function IterativeMethod with g{x) = {{x — 1)^ — 4)/2 one time 
and g{x) = (4 — (a; + 1)^) /2 another time: 

In [15]: xO, Eps = 1.0, le-8 

In [16]: g = lambda x: ((x+1) **2-4)/2.0 

In [17]: X, Iters = IterativeMethod(g, xO, Eps) 

In [18]: print('Approximate root is:’, x, ’ \nIterations :’, Iters) 
Approximate root is: -1.732050811416889 
Iterations: 59 

In [19]: g = lambda x: (4-(x-l) **2)/2.0 
In [20]: X, Iters = IterativeMethod(g, xO, Eps) 

In [21]: print('Approximate root is:’, x, ’ \nIterations :', Iters) 
Approximate root is: 1.732050811416889 
Iterations: 59 

Note: It is worthy to notice that the selection of the function g{x) is not 
unique. For example, for the function f{x) = — 3, the following iterative 

forms can be used: 

1. ^(n+l) = 4-(.W-l)^ 

2 . a.(n+l) ^ (3-a:W)a+a:W) 

3. 

The iteration Xn+i = cannot work. 


4.1.5 Using the MATLAB and Python solve Fnnction 

The MATLAB solve function can be used to find all the roots of a nonlin¬ 
ear equation f{x) = 0. The MATLAB function solve belongs to the symbolic 
toolbox. It finds the roots of f{x) = 0 analytically. 

To solve the nonlinear equations: 

1. x2-3 = 0 

2. e~^ = sina; 

3. -l-cosa; = Inx 

we can use the MATLAB commands: 

>> X = solve('x‘2 - 3’) 

X = 

3 -( 1 / 2 ) 
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-3''(l/2) 

>> X = solve(’exp(-x)-siii(x) ’) 

X = 

0.5885327439818610774324520457029 
>> X = solve (’x''3-cos(x)+log(x) ’ ) 

X = 

0.89953056480788905732035721409122 

In Python, we use the sympy. solve function to solve the nonlinear equa¬ 
tions: 

In [22]: from sympy import * 

In [23]: x = Symbol(’x’) 

In [24]: solve(x**2-3, x) 

[-sqrt(3), sqrt(3)] 

Note: The Python symbolic library does not look as mature as the MAT- 
LAB symbolic toolbox. The second and third problems cannot be solved with 
Python, but MATLAB solves them. Python raises the exception: 

In [25]: solve(exp(-x)-sin(x) , x) 

Traceback (most recent call last): 

File "<ipython-input-43-7d4dd4404520>", line 1, in <module> 
solve(exp(-x)-sin(x), x) 

raise NotImplementedError ( ’ \n’. join( [msg, not_impl_msg "/ f])) 
NotImplementedError: multiple generators [exp(x), sin(x)] 

No algorithms are implemented to solve equation -sin(x) + exp(-x) 

In [26]: solve(x**3-cos(x)+log(x)) 

Traceback (most recent call last): 

NotImplementedError: multiple generators [x, cos(x), log(x)] 

No algorithms are implemented to solve equation x**3 + log(x) - cos(x) 


4.2 Solving a System of Nonlinear Eqnations 

In this section, a System of linear equations of the form is considered: 

fl{xi,X2,...,Xn) = 0 

f 2 {xi,X 2 ,...,Xn) = 0 

fnixi,X2,...,Xn) = 0 

where, /i, /2, ■ • • >/n are nonlinear functions in the variables xi,X2, ■ ■ ■ ,Xn 
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We can write the above System in the vector form, by writing: 



/ a;i\ 



X = 

X2 

and f{x) = 

f2{x) 


\XnJ 


Kfn{x)j 


If a;* is any point in R”, the approximation by the first two terms of the 
Taylor expansion of f{x) around x* is given by: 

f{x) ~ f{x*) + J{x*){x-x*) 


where J{x*) is the Jacobian matrix and the ij component of the J is defined 
by: 




dftjx) 

dxj 


(a3=cc*) 


Now, if we set f{x) = 0, we obtain the equation: 


f{x*) + J{x*){x-x*) = 0 


from which we find that 


X = X* — J ^{x*)f{x*) 

Starting from an initial guess x^^') for the solution of f{x) = 0, the itera- 
tion: 

converges to the closest solution of f{x) = 0 to the initial point 

In MATLAB, the function 'jacobian’ can be used to find the Jacobian 
matrix of the nonlinear system of equations f{x). 

Example 4.1 Use MATLAB to find a solution to the nonlinear system of 
equations: 


a;2 + y2 = 30 

-a;2 + y2 = 24 


We write: 

f{x, y) = [x^ +y^- 30, - 24]'^ 

The Jacobian matrix for the nonlinear system is given by: 


J{x,y) 


2 x 2y 
-2x 2y 
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Starting from an initial guess , the iterative method is: 

^(n+l) ^ ^{n) _ J-1(2W)/(2W) 

_ _ f 2x(^'> 2yW\“Y \ 

+y(nf _ 24 ) 


The following MATLAB code implements the solution of the above given 
problem, starting from the initial guess 


1 % SolveWithJacobi . m 

2 function [z, Iterations] = Newton_sys(f, J, zO, Eps) 

3 iJ = @(z) inv{J(z)) ; 

4 z = zO - iJ(zO)*f(zO) ; 

5 Iters = 0 ; 

6 while norm(z-zO, inf) > Epsilon 

7 zO = z ; 

8 z = zO - iJ(zO)*f{zO) ; 

9 Iters = Iters + 1 ; 

10 end 


The execution of the above code, will give the results: 

» f = (§(z)[z(l)'2 + z(2)‘2 - 30; z(l)'2 + z(2)'2 - 24] ; 

» J = @(z) [2*z(l), 2*z(2); -2*z(l) , 2*z(2)] ; 

» zO = [1; 1] ; 

>> Eps = le-8 ; 

>> [z, Iterations] = Newton_sys(f, J, zO, Eps) ; 

>> fprintf (’Iterations = "/,i\n’, Iters) ; 

fprintf('The solution of the system is given by: \n\n\t\t\t\t 
xl = y.lS. 15f\n\t\t\t\t x2 = y.lS. 15f\n\n’, z(l) , z(2)) ; 
Iterations = 6 

The solution of the system is given by: 
xl = 1.732050807568877 

x2 = 5.196152422706632 

The Python code for the function SolveWithJacobi is: 


1 from numpy import array, inf 

2 from numpy.linalg import norm, inv 

3 def SolveWithJacobi(f, J, zO, Eps): 

4 iJ = lambda z: inv(J(z)) 

5 z = zO - iJ (zO) @f (zO) 

6 Iterations = 1 

7 while norm{z-zO, inf) > Eps: 

8 zO = z 

9 z = zO - iJ (zO) @f (zO) 

10 Iterations += 1 

11 return z, Iterations 
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In [27]: f = lambda z: array([z[0]**2+z[1]**2-30, 
-z[0]**2+z[l]**2-24]) 

In [28]: J = lambda z: array([[2*z[0] , 2*z[l]], [-2*z[0], 
2*z[l]]]) 

In [29]: Eps = le-8 

In [30]: zO = aj:ray([l., 1.]) 

In [31]: z, Iterations = SolveWithJacobi(f, J, zO, Eps) 

In [32]: printC’Approximate solution: (Iterations = 

Iterations,’) \nx =’, z [0], ’\ny =’, z[l]) 

Out[33]: Approximate solution: (Iterations = 7 ) 

X = 5.196152422706632 
y = 1.7320508075688774 

It is also possible to use the MATLAB function 'solve’ to find all the 
Solutions of /(a:) = 0, by using the MATLAB command: 

[xl, xl, ..., xn] = solve([fl, f2, ..., fn], [xl, x2, ..., xn]); 

For example: 

>> [x, y] = solve(’x~2+y'2-30’, ’-x'2+y'2-24’, [x y]) 

X = 

3'(1/2) 

3'(1/2) 

-3'(l/2) 

-3'(l/2) 

y = 

3*3'(1/2) 

-3*3'(1/2) 

3*3'(1/2) 

-3*3'(1/2) 

Using the sympy library, the problem can be solved through the following 
code: 

In [34]: from sympy import * 

In [35]: x, y = symbols(’x, y’) 

In [36]: solve( [x**2+y**2-30, -x**2+y**2-24], [x, y]) 

Out [37] : 

[(-sqrt(3), -3*sqrt(3)), 

(-sqrt(3), 3*sqrt(3)), 

(sqrt(3), -3*sqrt(3)), 

(sqrt(3), 3*sqrt(3))] 

Example 4.2 We use Newton’s method to solve the nonlinear system [47]: 


3a;i — cos 0 : 22:3 — 0.5 
x\ — 81 ( 2:2 + 0 . 1 )^ + sin 2:3 + 1.06 
IOtt-S 


^-X]_X2 , 


- 202:3 + 


0 

0 


3 


0 
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We write: 

/ = [ 3 x 1 — COSX 2 X 3 — 0.5,xi — 81(x2 + 0.1)^ + sinx 3 + + 20 x 3 + ——^—-]^ 


The Jacobian matrix associated to the given System is: 

( 3 X3sinx2X3 X2sinx2X3\ 

2xi —162x2 — 81/5 COSX 3 

-X 2 e-"^i "=2 -xie-"=i *2 20 / 

If Newton’s iteration starts from an initial condition , 

then Newton’s iteration is given by: 

^(n+l) ^ ^(n) _ 

The MATLAB code to compute the solution using the above iteration is 
given by: 

>> ciear ; clc ; 

>> f = @(z)[3*z(l)-cos(z(2)*z(3)-1/2 ; 
z(l)''2-81*(z(2)+0. l)''2+sin(z(3)) + 1.06 ; 
exp(-z(l)*z(2))+20*z(3)+(10*pi-3)/3] ; 

>> J = @(z) [3, z(3) *sin(z(2)*z(3)) , z(2) *siii(z(2)*z(3)) ; 

2*z(l), - 162*z(2) - 81/5, cos(z(3)); 

-z( 2 ) *exp(-z(l) *z( 2 ) ) , -z(l) *exp(-z(l) *z( 2 ) ) , 20 ] ; 

» zO = [0.1; 0.1; -0.1] ; 

>> Eps = le -8 ; 

>> z, Iterations = Newton_sys(f, J, zO, Eps) 

>> fprintf (’Iterations = "/iXn’, Iterations) ; 

Iterations = 4 

>> fprintf('The solution of the system is given by: \n\n\t\t\t\t xl = 
•/.18.15f\n\t\t\t\t x2 = "/.18.15f\n\t\t\t\t x3 = "/.18.15f\n’, zl(l) , zl(2), 
zl(3)) ; 

The solution of the system is given by: 
xl = 0.500000000000000 

x 2 = - 0.000000000000000 
x3 = -0.523598775598299 

Using Python, this example can be solved by calling the function 
Newton_sys as follows: 

In [38]: f = lambda z: array ( [3*z [0]-cos (z [1] *z [2] )-1/2, 
z [ 0 ] **2-81*(z [l]+ 0 . 1 ) ** 2 +sin(z [ 2 ] ) + l. 06, 
exp(-z [0] *z [1] )+20*z [2] +(10*pi-3) /3] ) 

In [39]: J = lambda z: array([[3, z [2] *sin(z [1] *z [2] ) , z [1] *sin(z [1] 
*z[2])], [2*z[0], - 162*z[l] - 81/5, cos(z[2])], 

[-Z [1] *exp(-z [0] *z [1] ) , -z [0] *exp(-z [0] *z [1] ) , 20]]) 

In [40]: zO = array([l., 1., 1.]) 

In [41]: z, Iterations = Newton_sys(f, J, zO, Eps) 
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In [42]: print('Approximate solution: (Iteratione = 
\nx =’, z [0] , ’\ny =’, z[l], ’\nz =’, z[2]) 
Approximate solution: (Iterations = 8 ) 

X = 0.49999999999999994 
y = -1.6530395442910908e-17 
z = -0.5235987755982988 


’ , Iterations,’) 
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Abstract 

Data interpolation means to use a given set of n +1 data points to approximate 
a function f{x) by a polynomial Pn{x) = anX^ + an-ix^~^ +.. . + aix + ao (of 
degree not exceedingn), such that Pn{xi) = f{xi), i = 0,...,n, where oq,■ ■ •,fln 
are constants. The data points are given by the table: 


X 

Xo 

Xi 


Xn 

fip 

fixo) 

fixi) 


fiXn) 


where Xi + Xj for z ^ j and xq < a^i < • • • < Xn- 

This chapter discusses some of the interpolation methods and their imple- 
mentation in MATLAB® and Python. It is divided into four sections. Section 1 
discusses Lagrange interpolation and its implementation in MATLAB and 
Python. Section 2 discusses Newton’s interpolation and the divided differ- 
ence technique for finding the coefficients of Newton’s interpolation. One- 
dimensional interpolations with MATLAB and Python are discussed in 
Sections 3 and 4. 


5.1 Lagrange Interpolation 

This section is divided into three sections. In the first section, construction 
and implementation of the Lagrange interpolating polynomial from a given 
data will be discussed. The proof of uniqueness of the Lagrange interpolating 
polynomial will be discussed in Section 2. In the last section, a formula of the 
interpolation error, using Lagrange interpolation will be presented. 

5.1.1 Construction of Lagrange Interpolating Polynomial 

In Lagrange interpolation, n+1 polynomials L(j{x),Li{x),...,Ln{x) are con- 
structed. Each polynomial Lj{x),{j = 0,... ,n) is of degree n, such that 

L^{xj) = l^ o’ 

105 
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Then, the interpolating polynomial Pn{x) is defined by the formula: 

n 

Pn{x) = Lo{x)yo + Li{x)yi + ... + Ln{x)yn = ^ L^{x)yi (5.1) 

i=0 

The polynomial Pn{x) is of degree n, because each Li{x) is an n*^-degree 
polynomial, for all i = 0,... ,n. Moreover, Pn{x) satisfies: 

n 

Pn{xj) = '^Li{Xj)yi = Lj{xj)yj = l-yj= y^, j = 0 ,...,n 
1=0 

This proofs that Pn{x) interpolates the given data. 

To construet the n + 1 polynomials Li(x), i = 0, ... ,n, it is important to 
notice that the roots of 


-li){x) = {x-Xo){x-Xi) ...{x-Xn) 
are xq^xi,. .. ,Xn- If L{x) is defined by 

L{x) = = {x-Xq){x-Xi) ...{x-Xi-i){x-Xi+i) . ..{x-Xn), 

X — Xi 

then Xi is not a root of L{x). 

Now, Li{x) can be obtained from L{x) using the formula: 


Li{x) = 


Li{x) _ {x-Xo){x-Xi)...{x-Xi-i){x-Xi+i)...{x-Xn) 


Li (xi) (xi-Xo)(Xi-Xl)... (xi -Xi-l)(Xi-Xi+l)... (xi - Xn) 

(5.2) 

The polynomials Li{x) can also be written as: 


n, ^7 

J=0 


and the Lagrange polynomial Pn{x) can be written as: 


^n(x)=^n 


X — Xj 

- -Vi 

Xi — Xj 


The following MATLAB function Lagrangeinterp.m constructs the 
Lagrange interpolation polynomial Pn{x) using the data points given be two 
vectors x and y, each of length n. It evaluates Pn{x) at x = t and returns the 
interpolation resuit. 


1 function p = Lagrangeinterp(t, x, y) 

2 n = length (x) ; 

3 p = 0 ; 
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4 for i = 1 : n 

5 s = 1 ; 

6 for j = 1 : n 

7 if j ^ i 

8 s = s* (t-x ( j) )/{x(i)-X(j) ) ; 

9 else 

10 continue 

11 end 

12 end 

13 p = p + s*Y (i) ; 

14 end 

Executing the above MATLAB code, we obtain: 

>> X = 0;pi/ll;pi ; 

>> y = sin(x) ; 

>> p = LagrangeInterp(pi/6, x, y) 

P = 

0.499999999914085 

A Python script Laginterp. py implements the Lagrange interpolating 
polynomials with the following code: 


1 # Laginterp.py 

2 import numpy as np 

3 def Lagrangeinterp(t, x, y): 

4 n = len (x) 

5 p = 0.0 

6 for i in range (n): 

7 s = 1 

8 for j in range (n): 

9 if j != i: 

10 s *= (t-x [ j ] ) / (x [i] -X [ j ] ) 

11 else: 

12 continue 

13 p += s*y[i] 

14 return p 

15 

16 X = np.linspace(0.0, np.pi, 11) 

17 y = np . sin (x) 

18 p = Lagrangeinterp(np.pi/6, x, y) 

19 print (p) 


Executing the code, shows the following results: 

runfile(’D:/PyFiles/LagInterp.py’, wdir=’D:/PyFiles’) 
0.4999999997868132 


5.1.2 Uniqueness of Lagrange Interplation Polynomial 

In this section the fundamental theorem of algebra will be used to prove that 
the Lagrange interpolating polynomial is unique. It States that any polynomial 
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of degree n with complex coefficients has at most n zeros. So, if a polynomial 
p{x) of degree n has more that n roots, it must be the zero polynomial p{x) = 
0,Va;G C. 

To prove the uniqueness of Lagrange interpolating polynomial, assume 
that Pn{x) and qn{x) are two polynomials of degree n, that interpolate the 
given data {xi,yi),i = 0,...,n. Then, pn{,Xi) = q-nixi) = yi for all i = 0,...,n. 
Let r{x) =Pn{x) — qn{x) for all x € [x(j,Xn]- Since both Pn{x) and qn{x) are 
both of degrees at most n, then so is r{x). Now, r{xi) = Pn{xi) — qn{xi) = 0 
for i = 0,...,n. That is r{x) has n + 1 roots in [xo,Xn], while it is of degree 
n. From the fundamental theorem of algebra, this cannot happen unless r{x) 
is the zero polynomial in [xo,Xn], that is r{x) = 0, Va; G [a:o,a;n]- This proves 
that Pn{x) = qnix), for x G [xo,Xn]- 

5.1.3 Lagrange Interpolation Error 

Given a function / : [a,5] — > R and n + 1 points xo,...,Xn in [a,b]. Let Pn{x) be 
a polynomial of degree n that interpolates the data (xq, f{xo )),..., {xn, f{xn))- 
that is pn{xj) = f{xj) for all j = 0,... ,n. 

From the Lagrange interpolation, Pn{x) is given by: 

n 

Pn{x) = '^Li{x)f{xi) 
i=0 

Define the interpolation error by: 

En{x) = f{x)-pn{x), X e[a,b] (5.3) 

Assuming that / is differentiable continuously, a formula for the function 
En{x) will be derived. 

The following theorem gives an estimate for Lagrange interpolation. 

Theorem 5.1 Let xq,xi,. .. ,Xn be n + 1 distinet values and t any real value. 
Let a = minjxo, ■ ■ -jXn} and b = maxjxo,.. -jXn}- Let f be a real valued func¬ 
tion on [a,b] that is differentiable n+1 times. Let Pn{x) be the Lagrange 
polynomial that interpolates the data (xq, f{xo)), ... ,{xn, f{xn)) ■ Then, there 
exists f G [a,b] such that 

En{x) = fix)-pn{x) = 

Proof: 

It is ciear that the resuit is true for a: = = 0,... ,n. Assume that the resuit 

is true when x + Xi. Fix a value t G [a,b] and define: 

G{x) = X G [a,5], 

where fi{x) = 
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Since the first n+1 derivatives exist for both and En{x), then so is 
G{x). Therefore, G{x) has n + 2 distinet roots in [a, 5]. From the intermediate 
value theorem, G'{x) has n + 1 distinet roots in (o,6) and G"{x) has n distinet 
roots in (a, b). By the mathematieal induetion, has n — k + 2 distinet roots 

in (a, 6). Henee, has at least one root in (a, 5). Let ^ be a root for 

in (a,5), that is G(’"+i)(e) = 0. 

From the definitions of En{x) and 'il){x), E^'^^\x) = and 

^(rt+i) = Then, 

g(-+i)(x) = 

W) 

At X = ^, 

G(-+i)( 5) = /("+!) (e) - = 0, 

W) 

from whieh, 


5.2 Newton’s Interpolation 

Given a set of data points (xj, f{xj)), j = 0,...,n, the problem is to find a 
polynomial Pn{x) of degree not exeeeding n sueh that Pn{xj) = f{xj) for all 
3 = 0,...,n. 

The Lagrange interpolating polynomial defined by 

n 

Pn{x) = '^Lj{x)f{xj), 
j=0 


where 


n 

Li{x) = 


1=1 

ji^t 


X — Xj 
Xi — Xj ’ 


suffer a problem that it is not possible to obtain Lj+i(x) from Li{x) (that is 
there is no iterative method to eompute the polynomials Li{x),i = 0,...,n). 
This makes the eomplexity of the algorithm high. Henee, Newton’s interpo¬ 
lating polynomials ean be seen as an alternati ve to the Lagrange polynomials. 


5.2.1 Description of the Method 

In Newton’s interpolation method, a polynomial Ffe(x) of degree k where 
1 < k < n is eomputed from Pk-i using an iterative teehnique. It starts with 
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a constant polynomial Po{x), where 

Po{x) =CQ = f{xo). 

Then, P^ is obtained from the iteration: 


fe-i 

Pk{x) = Pk-i{x) + CkY]_{x-Xj), (5.4) 

j=o 

where is a constant that is determined by substituting the data point 
{xk, f{xk)) in the equation of Pk{x), such that it satisfies the relation Pk{xj) = 
f{xj), j = 0,...,n. 

From the iterative formula of Newton’s interpolation polynomial, the 
Newton’s polynomial that interpolatos the data {xo, f{xo)),... ,{xk, f{xk)) 
descibed by equation (5.4) is of the form: 

Pkix) = Cq + Ci{x - Xo) + C2ix - Xo)ix - xi) -\ - \-Ck{x - Xq) . . . {x - Xk-l), 

(5.5) 

with Po{x) = co = f{xo). 

For a given data points {xj,f{xj)), the MATLAB function ComputeNewton- 
Coefs.m returns the coeficients of Newton’s polynomial. 


1 

functior 

c = ComputeNewtonCoefs(x, f) 

2 

n = 

length (x) ; 

3 

c = 

zeros (size (x)) ; 

4 

c(l) 

= f(i) ; 

5 

for 

j = 2 : n 

6 


z = X ( j) - X (1; j-1) ; 

7 


pml = c(1) ; 

8 


for k = 1 : j 

9 


pml = pml+sum (c(k) *prod (z(1:k-1))) ; 

10 


end 

11 


c(j) = (f (j)-pml) /prod {z (1: j-1) ) ; 

12 

end 


13 

end 



Another MATLAB function Newtoninterp.m calls the function Compute- 
Newtoncoefs to construet the corresponding Newton’s polynomial. It receives 
a set of data points {xj,f{xj)),j = 0,...,n and a value ^ to return the corre¬ 
sponding value yi obtained from the Newton’s interpolating polynomial. The 
code of the function Newtoninterp.m is: 


1 function yy = Newtoninterp(x, y, xi) 

2 c = ComputeNewtonCoefs(x, y) ; 

3 n = length (x) ; 

4 z = zeros (1, n) ; 

5 w = zeros { 1 , n) ; 

6 z(1) = 1 ; 

7 z(2:end) = xi - x(l:end-l) ; 
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8 


for k = 1 : n 


11 

12 end 


10 


9 


w(k) = prod (z(1:k)) ; 

end 

yy = sum (c.*w) ; 


To test whether the functions give true results, they have been tested to 
interpolate data sampled from the function 



A total of 10 equidistant data points {xj,f{xj)),j = 0,...,9 are used to gen¬ 
erate a Newton’s polynomial and then it is used to approximate the val- 
ues of the function at 101 equidistant data points. The MATLABs script 
NewtonInterpCos.m is used to do the task and plot the figure. Its code: 


1 ciear ; clc ; 

2 X = linspace(0, 2, 10) ; 

3 y = cos (4*x)./(1+x) ; 

4 XX = linspace(0, 2, 101) ; 

5 yy = zeros ( size (xx)) ; 

6 for j = 1 : length(yy) 

yy(j) = Newtoninterp (x, y, xx(j)) ; 

8 end 

9 plot (x, y, 'bo', XX, yy, '-.m’, 'LineWidth', 3) ; 

10 xlabel ( ' x ' ) ; 

11 ylabel ( ' y ' ) ; 

12 legend(’Data points (x, y) ', 'Newton Polynomial’) ; 

13 axis{[-0.1, 2.1, -0.8, 1.1]) ; 

14 set (gea, 'fontweight' , 'bold') ; 

15 grid on ; 

16 set (gea, 'XTick', 0:0.2:2) ; 

17 set (gea, 'YTick', -0.8:0.2:1) ; 

The resuit of exeeuting the code is shown in Figure 5.1. 

The corresponding Python functions ComputeNewtonCoef s . py and Newton¬ 
interp . py are implemented within a Python script NewtonInterpSin. py that 
interpolates 10 data points sampled from the function 


sin (4a;) 


f{x) 


Q<x<2 


1 + x 


to approximate the values of the function at 101 equidistant data points. The 
full code of NewtonInterpSin.py is: 


1 from numpy import zeros, prod, sin, linspace, arange 

2 import matplotlib.pyplot as plt 

3 def ComputeNewtonCoefs(x, f): 

4 n = len (x) 

5 c = zeros( len (x), 'float') 
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FIGURE 5.1: Approximat ion of the function cos (4x)/(l + a;) at 101 points, 
using a Newton’s interpolating polynomial constructed by 10 data points. 


6 z = zeros ( len (x), 'float') 

7 c[0] = f[0] 

8 for j in range(l, n): 

9 z[0] = 1.0 ; 

10 for k. in range(j) : 

11 z[k+l] = x[j]-x[k] 

12 pml = 0.0 

13 w = zeros(j, 'float') 

14 for k in range(j) : 

15 w[k] = prod ( z [ : k+1 ] ) 

16 pml += (c[k]*w[k]) 

17 c[j] = (f [j]-pml)/prod {z [: j + 1] ) 

18 return c 

19 

20 def Newtoninterp(x, y, xi): 

21 c = ComputeNewtonCoefs(x, y) 

22 n = len (x) 

23 z = zeros(n, 'float') ; 

24 w = zeros(n, 'float') ; 

25 z [ 0 ] = 1 ; 

26 for k in range(n-l) : 

27 z[k+l] = xi - x[k] 

28 for k in range(n) : 

29 w[k] = prod (z [: k+1 ] ) 

30 yy = sum(c*w) 

31 return yy 

32 

33 X = linspace(0, 2, 10) 

34 y = sin ( 4 *x) / (1+x) 

35 XX = linspace(0, 2, 101) ; 

36 yy = zeros ( len (xx), 'float') ; 

37 for j in range (len (yy)): 

38 yy[j] = Newtoninterp(x, y, xx[j]) ; 


39 
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40 plt.plot(x, y, 'bo', label='Data points (x, y) ' , Iw = 4) 

41 plt.plot(xx, yy, '-.m', label= 'Newton Polynomial', lw= 3) 

42 plt.xlabel( 'X' , fontweight= 'bold' ) 

43 plt.ylabel( 'y' , fontweight= 'bold’ ) 

44 plt. legend () 

45 plt.axis([-0.1, 2.1, -0,6, 1.0]) 

46 plt.grid(True, ls=':') 

47 plt.xticks(arange(0, 2,2, 0.2), fontweight= ’bold' ) 

48 plt.yticks(arange(-0.6, 1.0, 0.2), fontweight= 'bold' ) ; 

By executing the code, the data points and Newton’s interpolating poly- 
nomials are shown in Figure 5.2. 


5.2.2 Newton’s Divided Differences 

Given a data table: 


X 

Xo 

Xi 



flx) 

f{xo) 

f{xi) 


f{Xn) 


and the problem is to find a Newton’s polynomial 

n k — 1 

Pn{x) = co + ^Cfe 

k=l j=0 

that interpolates the given tabular data {xj,f{xj)), j = 0,... ,n. 

The basis functions for the Newton’s interpolating polynomials are the 
functions: 

No{x) = l,Ni{x) = x-xo,N 2 {x) = {x - xo){x - xi),... ,Nn{x) 

= (x-Xo) ...{x-Xn-l)- 



FIGURE 5.2: Approximation of the function sin(4x)/(l + x) at 101 points, 
using a Newton’s interpolating polynomial constructed by 10 data points. 














114 


Data Interpolation 


Therefore, computing a Newton’s polynomial is restricted to computing the 
polynomial coefficients cq, •..,c^- 

The method of divided differences is a short way for computing the coef¬ 
ficients of Newton’s interpolating polynomials Cj, j = 0, ... ,n. In a divided 
difference method, a divided differences table is constructed. 

Before discussing the methods of constructing Newton’s divided differences 
table, the following notations will be used: 


flxk] 

f ^Xk ^ X^ Xk-\-2\ 


f{xk) = yk, k = 0,...,n 
f[xk+i]-f[xk] 

^fc+1 


n—1 


^fc+2 ^k 


and generally, 


J [X]^ , , . . . , J - 

^k-\-j-\-l ^k 
k = 0,...,n-{j + l) 


Then, the Newton’s divided difference table is defined by: 


Xk 

f\xk\ 

f{Xk-l,Xk\ 


.f{xo,---,Xk\ 

Xo 

.f[xo\ 




Xi 

f[xi\ 









Xn 

f[Xn] 

f[Xn-l,Xn] 


f[xo,...,Xn] 


The coefficients of Newton’s interpolating polynomial are given by the 
following forms: 


co = f{xo) = f[xo] 

ci = -= /xo,a;i 

xi-xo 

f[xi,X2]- f[xo,Xi] 

C2 = - = f[xo,Xl,X 2 \ 

X2-X0 


and generally, 


^k 


f[xi,...,Xk]- f[xo,...,Xk-l] 
^k ^0 


f[xo,...,Xk],k= l,...,n. 


Hence, the coefficients of Newton’s interpolating polynomials are the diag- 
onal elements on the table of Newton’s divided differences. 
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A MATLAB function NewtonCoefs.m receives data points {xj,f{xj) and 
uses them to compute the Newton’s divided difference table and hence the 
coefficients of Newton’s interpolating polynomial. It returns both the divided 
difference table and coefficients. Its code is: 


1 function [ndd, Coef] = NewtonCoefs(x, f) 

2 % The function NewtonCoefs receives data points (x_j, ... 

f(x_j) and uses them 

3 % to compute the Newton's divided difference table and ... 

hence the 

4 % coefficients of Newton's interpolating polynomial. It ... 

returns both the 

5 % divided difference table and coefficients. 

6 Coef = zeros(l, length(x)) ; 

7 ndd = zeros (length (x), length(x)+l) ; 

8 ndd(:/ 1) = x' ; 

9 ndd , 2) = f' ; 

10 for j = 3 : length (x)+l 

11 for i = j-1 : length (x) 

12 k = j - 2 ; 

13 ndd(i, j) = (ndd(i, j-1)-ndd (i-1, ... 

j-l))/{x(i)-x(i-k)) ; 

14 end 

15 end 

16 for j = 1 : length (x) 

17 Coef{j) = ndd(j, j + 1) ; 

18 end 

19 end 

To test the performance of the function NewtonCoefs.m, five data points 

2 

{xj,f{xj)),j = 0,...,4: are considered, where f{x) = 4e“® sin(27rx), x S [0,1]. 
The following MATLAB instructions are used to do the test: 


>> X = linspace(0, 1, 5) ; 

» y = 4*exp(-x.~2).*sin(2*pi*x) ; 
>> [ndd, Coefs] = NewtonCoefs(x, y) 
ndd = 


0 

0 

0 

0 

0 

0 

0.2500 

3.7577 

15.0306 

0 

0 

0 

0.5000 

0.0000 

-15.0306 

-60.1224 

0 

0 

0.7500 

-2.2791 

-9.1165 

11.8282 

95.9341 

0 

1.0000 

-0.0000 

9.1165 

36.4661 

32.8506 

-63.0836 

Coef = 







0 15.0306 -60.1224 95.9341 -63.0836 


The Newton’s interpolating polynomial obtained by using these coefficients 
is shown in Figure 5.3. 
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2 

FIGURE 5.3: Approximation of the function 4e“® sin(27ra:) at 101 points, 
using a Newton’s interpolating polynomial constructed by 5 data points. 


5.3 MATLAB’s Interpolation Tools 

MATLAB provides many functions for data interpolation. In this section, the 
MATLAB functions interpl, spline and pchip will be discussed. 

5.3.1 Interpolation with the interpl Fnnction 

The MATLAB function interpl receives mainly two vectors x and y and 
a third argument which could be a scaler value or a vector. In this case, it 
interpolates the data {xj,yj) with piecewise linear polynomials. To see this, 8 
data points will be sampled from the function y = e“^sin(a:) in the interval 
[0,3], then the constructed interpolating polynomial will be evaluated at 100 
data points in [0,3]. This is can be done through the following MATLAB 
commands: 

To construet the sample data points: 

>> X = linspaceCO, 3, 8) ; 

>> y = exp(-x).*sin(x) ; 

To define the points s at which the interpolating polynomial will be evaluated: 
>> s = linspaceCO, 3) ; 

The function interpl is used as follows: 

>> P = interpl(x, y, s) ; 

The function interpl joins any two points {xj,yj) and (ccj-i-i,j/j+i) by 
straight lines. Hence, the interpolating polynomial is a piecewise-linear 
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O Data (X, y) 

■ ■ ■ ■ Interpolating Polynomlal y^sP^{s) 



0 0.3 0.6 0.9 1.2 1.S 1.6 2.1 2.4 2.7 3 

X 

FIGURE 5.4: The original data {x^y) and the data obtained by interpolation 
{s,P{s)). 

polynomial consisting of the union of the straight lines joining between the 
points {xj,yj), j = 0,... ,7. At any given point s/j,fc = 0,... ,99 the polynomial 
is evaluated by determining i such that Xi < s < Xi^i and using the equation 
of the line joining between (xi,yi) and (xi-^.l,yi-^-l). It finally returns the resuit 
in P. 

In MATLAB the data and the interpolating polynomial are plotted by 
using: 

>> plot(x, y, ’ro’, s, P, ’:b’, ’LineWidth’, 3) ; 

>> xlabel(’x’) ; 

>> ylabel(’y’) ; 

>> legendC’Data’, ’(s, P(s))’) 

The data points and interpolating polynomial are plotted in Figure 5.4. 

5.3.2 Interpolation with the Spline Function 

Interpolating with piecewise linear polynomials resuit in functions that are not 
differentiable at the interpolation points as seen in Figure 5.4. The MATLAB 
function spline computes a piecewise cubic polynomial such that the inter¬ 
polating cubic polynomial and its first and second derivatives are smooth at 
the interpolation points {xj,yj). 

If the command interpl is replaced by the command spline in the pre- 
vious example, and replot the resuit: 

>> P = splineCx, y, xx) ; 

>> plotCx, y, ’ro’, XX, P, ’:b’, ’LineWidth’, 3) ; 

>> xlabel(’x’) ; 

>> ylabel(’y’) ; 

>> legend(’The data (x, y)’, 'The spline P(x)’) 
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The Figure 5.5 will be seen: 



' 

' 

' 


' 

1 0 The data (x, y) 1 











\ 



- 







- 







- 
















0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7 


FIGURE 5.5: The original data {x,y) and the data obtained by interpolation 
{xx,P{xx)). 

By typing: 

>> Q = splineCx, y) 

The resuit is: 

Q = 

form: ’pp’ 

breaks: [0 0.4286 0.8571 1.2857 1.7143 2.1429 2.5714 3] 

coefs: [7x4 double] 

pieces: 7 

order: 4 

dim: 1 

The structure Q contains information about the interpolating polynomial. 
For 8 data points, 7 pieces of cubic splines are required to join between each 
two data points {xi,yi) and {xi-^-l,yi-^-l);i = 0,... ,7. The field “breaks” con¬ 
tains the X — coordinates of the data points. The field “coefs” contain the 
coeficients of each cubic spline piece between the two data points {xi,yi) and 
{xi-^-l,yi-^-l);i = 0,...,7. The error in approximating the original function by 
the interpolating spline polynomial is h^, where h = maxa;j_|_i — Xi\i= 1,..., 7. 
Therefore, the order of convergence is 4. 

Then, Q can be evaluated at the points of the vector xx by using the ppval 
function. This is done as follows: 


>> S = ppval(Q, xx) ; 
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Now, S and P are identical. This can be seen through using the command: 

>> Error = norm(P-S, inf) 

Error = 

0 

5.3.3 Interpolation with the Function pchip 

The function pchip stands for piecewise cubic Hermite interpolating polyno- 
mials. It employs sets of the cubic Hermite polynomials to approximate the 
function between any pair of data points. 

The function pchip works in a similar way to the function spline. If the 
command spline is replaced by the command pchip in the previous example, 
by typing: 

>> P = pchipCx, y, xx) ; 

>> plot(x, y, ’ro’, XX, P, ’;b’, 'LineWidth’, 2) ; 

>> xlabel(’x’) ; 

>> ylabel(’y’) ; 

>> legend(’The data (x, y)’, 'The pchip P(x)’) 

then Figure 5.6 will be obtained. 

Again, if Q is defined by: 

>> Q = pchip(x, y) 

the following resuit would be obtained: 

Q = 

form: ’pp’ 


O The data (x, y) I 

. Hermite cubic spline P(x) | 


0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7 


FIGURE 5.6: The original data (a;,y) and the data obtained by interpolation 
{xx,P{xx)) using the pchip function. 
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breaks; [0 0.4286 0.8571 1.2857 1.7143 2.1429 2.5714 3] 

coefs; [7x4 double] 

pieces; 7 

order; 4 

dim; 1 

It is similar to what was obtained by using the spline command. The 
difference is in the values of the coefhcients, and therefore, in the equations 
defining the 7 pieces of the interpolating polynomial. 

The function ppval can be used to evaluate the interpolating polynomial 
Q at the components of xx. 

» S = ppval(Q, xx) ; 

» Error = norm(P - S, inf) 

Error = 

0 

5.3.4 Calling the Functions spline and pchip from interpl 

The MATLAB function interpl can perform the interpolation by using the 
cubic splines or the piecewise cubic Hermite polynomials instead of using a 
piecewise linear interpolation. 

The MATLAB command: 

>> P = interpl(x, y, xx, 'spline’) ; 

will interpolate the data {x,y) by piecewise cubic splines similar to the com¬ 
mand spline does, and the MATLAB command: 

>> P = interpl(x, y, xx, 'pchip’) ; 

will interpolate the data {x,y) by piecewise cubic Hermite polynomials as same 
as the command pchip does. 

If the MATLAB command: 

>> P = interpl(x, y, xx, 'cubic') ; 

is executed, then it can do the same job as the function pchip. 


5.4 Data Interpolation in Python 

The Python’s scipy. interpolate library contains various kinds of functions 
for data interpolation [18]. Some of these functions are CubicHermiteSpline, 
CubicSpline, LinearNDInterpolator,NearestNDInterpolator,interpld, 
lagrange, pade, pchip, pchip_interpolate. There are many other func¬ 
tions for multiple dimension interpolation. In this section some of the Python’s 
interpolation functions will be discussed. 
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5.4.1 The Function interpld 

The Python’s function interpld has a similar syntax to the MATLAB func¬ 
tion interpl. If no kind of interpolation is received, it interpolatos the 
given data points with linear piecewise polynomials. If it receives the kind 
(quadratic, cubic, etc.), then it interpolatos the given data with that kind of 
piecewise continuous polynomial. 

For example, ten equidistant sample points from the function 

/(,) = 

l + x 

will be used to interpolate the function with linear, quadratic and cubic 
piecewise polynomials, based on the interpld function. The Python script 
interpwpld.py implements the interpolation with interpld: 


1 from numpy import cos, linspace, arange 

2 from scipy.interpolate import interpld 

3 import matplotlib.pyplot as plt 

4 X = linspace(0, 3, 10) 

5 y = cos(4*x)/(l+x) 

6 XX = linspace(0, 3, 101) ; 

7 Linld = interpld(x, y) 

8 Quadld = interpld(x, y, kind = 'quadratic') 

9 Cubicld = interpld(x, y, kind = 'cubic') 

10 yyl = Linld (xx) 

11 yy2 = Quadld (xx) 

12 yy3 = Cubicld (xx) 

13 

14 plt. subplot (211) 

15 plt.plot(x, y, 'bo', label='Data points (x, y) Iw = 4) 

16 plt.plot(xx, yyl, ls='-.', color= 'orangered' , label= 'Linear' , ... 

lw= 2) 

17 plt.plot(xx, yy2, ls=':', color= 'purple' , label= 'Quadratic' , ... 

lw= 2) 

18 plt.xlabel ( 'X' , fontweight= 'bold' ) 

19 plt.ylabel( 'y' , fontweight= 'bold' ) 

20 plt. legend () 

21 plt.grid(True, ls=':') 

22 plt.xticks(arange(0, 3.3, 0.3), fontweight= 'bold' ) 

23 plt.yticks(arange(-0.6, 1.2, 0.2), fontweight= 'bold' ) 

24 

25 plt.subplot (212) 

26 plt.plot(x, y, 'bo', label='Data points (x, y) ', Iw = 4) 

27 plt.plot(xx, yy3, ls='--', color= 'crimson' , label= 'Cubic' , lw= 2) 

28 plt.xlabel( 'X' , fontweight= 'bold’ ) 

29 plt.ylabel (' y' , fontweight= 'bold’ ) 

30 plt. legend () 

31 plt.grid(True, ls=':') 

32 plt.xticks(arange(0, 3.3, 0.3), fontweight= 'bold' ) 

33 plt.yticks(arange(-0.6, 1.2, 0.2), fontweight= 'bold' ) 


Executing this code shows in the graphs in Figure 5.7. 
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• Data points (x, y) 
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0.0 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7 3.0 


FIGURE 5.7: Interpolation of 10 equidistant sample points taken from f{x) = 
cos(4a:)/(l + x),0 < x < 3, with linear, quadratic and cubic polynomials. 


5.4.2 The Functions pchip_interpolate and CubicSpline [18] 

The two functions pchip_interpolate and CubicSpline have the same func¬ 
tions as the MATLAB’s functions pchip and spline. They receive three 
arguments {x,y,ps), the first two arguments represent the interpolation data 
points, and the last argument is the set of points at which the function is to 
be approximated. 

The Python code interppchipspline .py interpolatos the function f{x) = 
sin(4a;)/(l-|-a;) at 10 data points in [0.0,3.0] and uses the interpolation poly- 
nomial to approximate the function at 101 data points. 


1 from numpy import sin, linspace, arange 

2 from scipy.interpolate import pchip.interpolate, CubicSpline 

3 import matplotlib.pyplot as plt 

4 X = linspace(0, 3, 10) 

5 y = sin ( 4 *x)/ ( 1+x) 

6 XX = linspace(0, 3, 101) ; 

7 

8 yyl = pchip.interpolate (x, y, xx) 

9 yy2 = CubicSpline(x, y)(xx) 
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FIGURE 5.8: Interpolation of 10 equidistant sample points taken from 
f{x) = sin{Ax)/{1 + x),0 < a; < 3, with the functions pchip_interpolate and 
CubicSpline. 


10 

11 plt.plot(x, Y, 'bo', label='Data points (x, y) Iw = 4) 

12 plt.plot(xx, yyl^ ls='-', color= 'orangered' , ... 

label= 'pchip.interpolate ' / lw= 2) 

13 plt.plot(xx, yy2, ls='--', color= 'purple' , ... 

label= 'CubicSpline' , lw= 2) 

14 plt.xlabel( 'X' , fontweight= 'bold’ ) 

15 plt.ylabel( 'y' , fontweight= 'bold’ ) 

16 plt. legend () 

17 plt.grid(True, ls=':') 

18 plt.xticks(arange(0, 3.3, 0.3), fontweight= ’bold' ) 

19 plt.yticks(arange(-0.6, 1.2, 0.2), fontweight= 'bold' ) 


Executing the code shows in Figure 5.8. 


5.4.3 The Function lagrange 

The Python interpolation function lagrange interpolates a given data points, 
based on Lagrange interpolation. It receives two vectors {x,y) representing 
the coordinates of the interpolation points to create a polynomial of degree n, 
where n + 1 is the number of interpolation points. 

The Python script Lagrangeinterp.py uses Lagrange interpolation to 
construet a ninth degree polynomial that interpolates data sampled from the 
function f{x) = {cos{bx) + sin{bx))/{2{1 + x"^)) based on 10 data points in 
[0,3]. 


1 from numpy import sin, cos, linspace, arange 

2 from scipy.interpolate import lagrange 

3 import matplotlib.pyplot as plt 

4 X = linspace(0, 3, 10) 
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5 y = {sin(5*x) +COS (5*x))/(2*(l+x**2)) 

6 XX = linspace(0, 3, 101) ; 

7 LG = lagrange(x, y) 

8 print (LG) 

9 yyl = LG(xx) 

10 

11 plt.plot(x, y, 'bo', label='Data points (x, y) markersize = 8) 

12 plt.plot(xx, yyl, ls='-'/ color= 'purple' , label= 'Lagrange' , ... 

lw= 2) 

13 

14 plt.xlabel ( 'X' , fontweight= 'bold’ ) 

15 plt.ylabel( 'y' , fontweight= 'bold' ) 

16 plt. legend () 

17 plt.grid(True, ls=':') 

18 plt.xticks(arange(0, 3.3, 0.3), fontweight= ’bold' ) 

19 plt.yticks(arange(-0.6, 1.2, 0.2), fontweight= 'bold' ) 

By running the code, the following output is obtained: 

runfile(’D:/PyFiles/Lagrangelnterp.py’, wdir=’D:/PyFiles’) 

9 8 7 6 5 4 3 

- 0.4591 X + 6.308 x - 35.79 x + 107.5 x - 180.9 x + 164 x - 66.05 x 
2 

+ 2.858 X + 1.864 x + 0.5 

Figure 5.9 shows the graph of the function f{x) approximated at 101 points 
using Lagrange interpolation. 



X 


FIGURE 5.9: Lagrange interpolation the function f{x) = {cos{5x) + 
sin{5x))/{2{1 + x"^)) based on 10 data points in [0,3]. 
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Numerical Differentiation and Integration 


Abstract 

This chapter discusses the numerical methods for approximating derivative 
and integrations of functions. The chapter is divided into two sections: the 
first section discusses the numerical differentiation of functions based on 
finite difference formulas and the second discusses the numerical integra¬ 
tion based on Newton-Cotes and Gauss methods. Such numerical differen¬ 
tiation or integration algorithms are implemented using both MATLAB® and 
Python. 


6.1 Numerical Differentiation 

6.1.1 Approximating Derivatives with Finite Differences 

Let xq G (a,b) C R and / G C"+^[a,fo]. Then, for /i > 0 and k < n the Taylor 
expansion of / around xq is given by: 

f(xo + h) = f(xo) + hf'(xo)+ ^f"(xo)-l -^ 

tfe+i 

+ ^ [xo,xo-ffe] (6.1) 

and 

f(xo + h) = f(xo) - hf'(xo) + ^f"(xo) -+ (a^o) 

+ (k + 1)! [xQ-h,xo\ (6.2) 
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By setting fc = 1 in Equation (6.1) and solving for f'{xo) the first derivative 
of f{x) at X = xq is given by: 

^ f{xo + h)- f{xo) , , Li fr o\ 

f i^o) — -^ [^0)2:o + ^] (6.3) 

Hence, by taking as small possible value of ft. > e > 0, f'{x) can be approxi- 
mated by: 

/'(x„) = + (6.4) 

h 

where e denotes the machine precision. If h is taken to be less than £, round-off 
error can affect the accuracy of approximate derivative at xq. 

The approximate formula of the derivative f' (x)' in Equation (6.4) is called 
the forward difference formula. 

By setting fc = 1 in Equation (6.2) and solving for /'{xq) the first derivative 
of f{x) at X = Xq is given by: 

fixo) = G N - h,xo] (6.5) 

from which f^{x) can be approximated by: 

^ fi^« + k)-fM ,6 6 ) 

h 

where e denotes the machine precision. If h is taken to be less than e, round-off 
error can affect the accuracy of approximate derivative at xq. 

The approximate formula of the derivative f {x)' in Equation (6.4) is called 
the forward difference formula. Also, the approximate formula of fix) 
in Equation (6.6) is called the backward difference formula [31]. 

The last terms of equations (6.3) 

Riix) = ^f'i^)’ 2;G [xo,xo + h] 

and (6.6) 

R 2 it) = -^f"it), tG[xo-h,xo] 

give the remainders at points x G [xo,xo + h] and xo — h,xo, respectively. 
Ignoring the remainder while approximating some function is known as the 
truncation error. If f"ix) is bounded by a constant Mi G in [to,to + h] 
and by M 2 G R+ in [xo — h,xo], then: 

|.R2(^)I — Mi-h,forxG[xo,xo + h] (6.7) 

|i? 2 (^)l < M 2 -h,for tG[xo — h,xo]- (6.8) 


This indicates that the truncation error in both the forward and backward 
difference formulas are Oih). 
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Example 6.1 In this example, Equation (6.4) will be used to find approxi- 
mation of derivative of f{x) = )/^ for h = 10“^,..., 10“^®. The exact 

derivative of f{x) is 

The error for each value of h will also be shown. 

The Python code is: 


1 #fpapproxim.py 

2 import numpy as np 

3 f = lambda x: np.exp(-np,sin(x**3)/4) 

4 fp = lambda x: -3*x**2*np.cos(x**3)/4*f(x) 

5 h = 0.1 

6 fpapprox = [] 

7 Eps = np.spacing(1.0) 

8 while h > Eps: 

9 fpl = (f(l.+h)-f(1.))/h 

10 fpapprox.append([h, fpl, np. abs (fp(1.)-fpl)]) 

11 h /= 10 

12 print ( '-' ) 

13 print ( ' h', '\t\t\t', 'Approx Der', '\t\t 'Approx Error') 

14 print ( '-' ) 

15 for X in fpapprox: 

16 print (' {0:1,3e} format (x[0]), ' \t' , ... 

'{0:1.15e}' . format (x[l]), ’\t\t', ... 

'{0:1.15e}' . format (x[2])) 


Executing this code will give the results: 

runfile(’D:/PyFiles/fpapproxim.py’, wdir=’D:/PyFiles’) 


h Approx Der Approx Error 


l.OOOe-01 

l.OOOe-02 

l.OOOe-03 

l.OOOe-04 

l.OOOe-05 

l.OOOe-06 

l.OOOe-07 

l.OOOe-08 

l.OOOe-09 

l.OOOe-10 

l.OOOe-11 

l.OOOe-12 

l.OOOe-13 

l.OOOe-14 

l.OOOe-15 


-2.589437504200143e-01 
-3.231226286809608e-01 
-3.278426595997308e-01 
-3.282990900810301e-01 
-3.283445787927164e-01 
-3.283491261107940e-01 
-3.283495819683679e-01 
-3.283496252670658e-01 
-3.283496807782170e-01 
-3.283495697559146e-01 
-3.283484595328899e-01 
-3.284039706841212e-01 
-3.286260152890463e-01 
-3.330669073875469e-01 
-4.440892098500626e-01 


6.940588094341621e-02 
5.227002682469728e-03 
5.069717636996818e-04 
5.054128240039590e-05 
5.052570714092486e-06 
5.052526365068033e-07 
4.939506259571402e-08 
6.096364635332918e-09 
4.941478654041376e-08 
6.160751592210190e-08 
1.171830540547258e-06 
5.433932069076608e-05 
2.763839256157974e-04 
4.717276024116424e-03 
1.157395784866321e-01 
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The MATLAB code is: 


1 %fpapproxim . m 

2 f = @(x) exp (-sin (x"3)/4) ; 

3 fp = @(x) -3*x"2*cos(x"3)/4*f(x) ; 

4 fpapp = zeros(15, 3) ; 

5 h = le-1 ; 


6 j = 1 ; 

7 while h > eps 

8 fpl = (f{1.0+h)-f(1,0))/h ; 

9 fpapp(j,:) = [h, fpl, abs (fp (1) -fpl) ] ; 

10 h = h / 10 ; 

11 j = j + 1 ; 

12 end 

13 fprintf ( '-\n ' ) ; 

14 fprintf ( ' h \t\t\t\t Approx Der \t\t\t Approx Error\n’) ; 

15 fprintf ( '-\n ' ) ; 

16 for j = 1 : 15 


17 fprintf (' %5.3e\t\t%l6.15e\t\t%16.15e\n' , fpapp(j,l), ... 

fpapp(j,2), fpapp(j,3)) ; 

18 end 

Executing the MATLAB code gives similar results as Python code. 

Figure 6.1 shows the relationship between the value of h and the corre- 
sponding error. It shows that taking smaller value of h improves the derivative 
approximation up to some limit, after that the approximate derivative gets 
worse. 

The implementation of a Python/MATLAB code to approximate f'{l) 
using formula (6.6) is by replacing the line: 

fpl = (f(1.0+h)-f(1.0))/h 



FIGURE 6.1: Approximation errors of /'(1) versus different values of h in the 
loglog scale, using the forward difference formula 6.4. 
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by the line: 

fpl = (f(l.O)-f(1.0-h))/h 
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in the codes of Example (6.1). Similar results can be obtained by running a 
code fpbdfapprox.py that is based on Equation (6.6): 

runfile(’D:/PyFiles/fpbdfapprox.py’, wdir=’D:/PyFiles’) 
h Approx Der Approx Error 


l.OOOe-01 

l.OOOe-02 

l.OOOe-03 

l.OOOe-04 

l.OOOe-05 

l.OOOe-06 

l.OOOe-07 

l.OOOe-08 

l.OOOe-09 

l.OOOe-10 

l.OOOe-11 

l.OOOe-12 

l.OOOe-13 

l.OOOe-14 

l.OOOe-15 


-3.631033359769975e-01 
-3.332305430411631e-01 
-3.288531422600549e-01 
-3.284001380376989e-01 
-3.283546835874951e-01 
-3.283501365247687e-01 
-3.283496807782171e-01 
-3.283496363692961e-01 
-3.283495697559146e-01 
-3.283495697559146e-01 
-3.283484595328899e-01 
-3.282929483816587e-01 
-3.275157922644211e-01 
-3.219646771412953e-01 
-2.220446049250313e-01 


3.475370461356703e-02 
4.880911677732580e-03 
5.035108966244262e-04 
5.050667426836908e-05 
5.052224064605593e-06 
5.051613382045517e-07 
4.941478659592491e-08 
5.005865610918647e-09 
6.160751592210190e-08 
6.160751592210190e-08 
1.171830540547258e-06 
5.668298177174957e-05 
8.338390990093592e-04 
6.384954222135142e-03 
1.063050264383992e-01 


By setting fc = 2 in equations (6.1) and (6.5), subtracting Equation (6.5) 
from (6.1) and solving for f'{x) gives: 

H, f{xo + h)- f{xo-h) ^ h , m n^ 

f(xo) = - ^ -hy/ [t), T e[xo-h,xo + h] (6.9) 

from which a third formula of the derivative is obtained 

/'(X.) = + (6^10) 

The formula in Equation (6.10) is called the Central difference formula. 
If \f”'{x) \ < M G R+, then the approximation error of the Central difference 
formula is bounded by M/3 • h? which means that the Central difference for¬ 
mula is 0{h‘^) method. 

The Python/MATLAB code for implementing the Central difference for¬ 
mula, is obtained by replacing the line: 

fpl = (f(1.0+h)-f(1.0))/h 

in Example 6.1, by the line: 

fpl = (f(1.0+h)-f(1.0-h))/(2*h) 
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FIGURE 6.2: Comparing the forward differencing errors to the Central differ- 
ence errors in the loglog scale. 


In Figure 6.2 the approximation errors obtained by the forward difference 
formula are compared to those obtained by the Central difference formula in 
the loglog scale, for different values of h. 

From Figure 6.2 it is ciear that the Central difference formula has better 
approximation of the first derivative than the forward difference formula, since 
it produces smaller errors. 

In Figure 6.3 the graph of exact derivative of f{x) = ) is plotted 

in the interval [0,7r/2] against the graph of approximate derivative obtained 
by a forward difference formula. 

If the interval [0.0,7r/2] is partitioned into N sub-intervals [a:o,a:i],..., 
[a;jv-i)a^Ar], then the Python function diff in the numpy library can be used 
to compute a vector df = [f{xi)-f{xo),f{x 2 )-f{xi),...,f{xN+i)-f{xN)]. 
If dx = tt/(2N), then df/dx returns approximate values of the derivatives 
at the points xq, ... ,X]\[-i. The following Python commands use the numpy 
function diff to graph f'{x): 


1 import numpy as np 

2 import matplotlib 

3 import matplotlib.pyplot as plt 

4 matplotlib.rc (' text' , usetex=True) 

5 matplotlib.rcParams[ 'text.latex.preamble'] ... 

= [r''\usepackage{amsmath} " ] 

6 a, b= 0.0, 2.0 

7 N = 1000 

8 X = np.linspace(a, b, N+1) 

9 f = lambda x: np.exp(-np,sin(x**3)/4) 

10 df = np . dif f {f (x) ) 
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FIGURE 6.3: Graph of exact derivative of f{x) = is plotted in the 

interval [0,7r/2] against the graph of approximate derivative obtained by a 
forward difference formula. 


11 dx = (b-a) /N 

12 dfdx = df/dx 

13 plt.plot(x[:N], dfdx, '-m', ... 

lw=2, label = r ' $\mathbf{-\frac{3x''2\cos (x''3)}{4} ... 
e~{-\frac{\sin(x"3)}{4}}}$') 

14 plt.xlabel ( 'X' , fontweight= 'bold’ ) 

15 plt.ylabel( 'y' , fontweight= 'bold’ ) 

16 plt.grid(True, ls=':') 

17 plt.legend(loc= 'upper left') 


Figure 6.4 shows the graph of f'{x),x S [0, using the function diff. 



FIGURE 6.4: Graph of f'(x),x G [0, |] using the function diff. 
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To find an approximation of f"{x) at x = xq, Equation (6.1) is added to 
(6.5) and solving the resulting quantity for giving: 

^ fixo-h)-2f{xo)+fixo + h) L ,1.1 

/ (x) = —--— ^2 —12'^ ^ ^ Fo-/i,xo + /iJ 

from which, 

f" (x) ~ ~ + /(^0 + (g 

If \f^^\x) \ < M G R+, then ^\f^"^\x) \ < hence the truncation error in 

Equation (6.11) is of 0{K^). 

The second derivative of /(x) = is f''{x) = 2(1 —2x^)e“®^. The follow- 
ing MATLAB commands approximate the second derivative of /(x),x S [—3,3] 
and graph it. 

>> X = linspace(-3, 3, 601) ; 

>> f = @(x) -exp(-x.“2) ; 

>> y2 = diff(f(x), 2) ; 

>> plot(x(2:end-1), y2, ’-b’, ’LineWidth’, 2) 

>> xlabel(’x’) ; ylabel(’y’) ; 

>> legendC’d"2f(x)/dx"2 = 2(l-2x"2)e'{-x"2}’) ; 

>> grid on 

In Figure 6.5 the graph of f"{x) = 2(1 —2x^)e“®^ is plotted in the interval 
[-3,3]. 

It is very important to notice that if h? < e the resulting approxima¬ 
tion errors get too large. For example, if we let h take the values 10“-^, 



FIGURE 6.5: Graph of f"{x) = 2(1 —2x^)e * in the period [—3,3]. 
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j = the corresponding approximate values and errors of f"(x) will 

be as follows: 


h 


Approx Der 


Approx Error 


l.OOOe-01 

l.OOOe-02 

l.OOOe-03 

l.OOOe-04 

l.OOOe-05 

l.OOOe-06 

l.OOOe-07 

l.OOOe-08 

l.OOOe-09 

l.OOOe-10 

l.OOOe-11 

l.OOOe-12 


-7.296463309943767e-01 
-7.356975709826852e-01 
-7.357582692546494e-01 
-7.357588760470435e-01 
-7.357581210953866e-01 
-7.358003095703222e-01 
-7.271960811294772e-01 
-1.665334536937734e+00 
1.110223024625156e+02 
5.551115123125780e+03 
5.551115123125780e+05 
1.110223024625156e+08 


6.112551348507966e-03 
6.131136019948968e-05 
6.130882352906042e-07 
6.295841181724882e-09 
7.612474980378536e-07 
4.142722743749605e-05 
8.562801213407467e-03 
9.295756545948495e-01 
1.117580613448585e+02 
5.551850882008122e+03 
5.551122480714604e+05 
1.110223031982745e+08 


The table above shows that as the step size h gets less than 10“®, the error 
becomes larger and larger. This is because of the approximation formula 
requires division by < 10“^® < e. 


6.2 Numerical Integration 

If / is a function that is defined over a real interval [a,b], then the area under 
the curve of / from x = a to x = & is given by the definite integral: 

1= [ f{x)dx 


If F(x) is the anti-derivative of /(x) in [a,b], then from the fundamental 
theorem of calculus 


1= f f{x)dx = F{b) — F{a) 

J a 


However, there are many functions defined over finite intervals whose anti- 
derivatives are unknown, although they do exist. Such examples include: 


h = 


sin (x) 


dx 


and 


I2 = 


dx 
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The numerical integration methods work to find definite integral of a func- 
tion / in an interval through first partitioning the interval [a,b] by points 
xo,Xi,...,X]s[, such that a < xq < Xi < ... < xn < b, then approximate / by a 
sum of the form: 

rb 


.0 N 

1=1 f{x)dxPi^Wjf{xj) 
j=0 


( 6 . 12 ) 


where the different numerical integration methods differ from each other in 
the way by which the points Xj are selected and coefficients Wj are calculated. 

Equation (6.12) is called numerical quadrature. Hence, the term numer¬ 
ical quadrature points to a form, in which an integration formula is approxi- 
mated by a finite sum. 

In this chapter, we discuss two classes of numerical integration methods, 
namely: the Newton-Cotes and Gauss quadratures. 

6.2.1 Newton-Cotes Methods 

In Newton-Cotes methods, the points xq,...,xn are chosen such that xo = a < 
xi < ... < Xn = b, with h= (b — a)/n. The idea behind Newton-Cotes methods 
to approximate I = jl^f{x)dx is first to approximate /(x) by a Lagrange 
polynomial Pn{x) of degree n. Then, to approximate I by the integral of 
Pn{x) [43, 47]. That is: 

ph pb pb ^ \ 

/ f{x)dxKi / Pn{x)dx= / '^Lj{x)f{xj)dx = '^i / Lj{x)dx \ f{xj) 

Ja Ja Ja ^^0 V J 

(6.13) 

Hence, the quadrature weights Wj in Newton-Cotes methods are given by: 


Wj = / Lj{x)dx 
J a 


The numerical integration formulas for n = 1 and n = 2 will be derived and 
discussed. 

(I) n = 1: when n = 1, h= {b—a)/l = b — a and the given data points are 
(a,/(a)) and (6,/(6)). The Lagrange interpolating polynomial is 

hence, 

J f{x)dxKi^j [-{x-b)f{a) + {x-a)f{b)]dx=^{f{h) +f{a)) 

(6.14) 
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The formula in equation (6.14) gives the trapezoidal rule, where the 
area under the f{x) curve from x = a to x = b is approximated by area 
of a trapezium whose bases are /(a) and f{b) and height is b — a. 

In Figure 6.6 the area under the curve of f{x) is approximated by the 
area of the trapezium (shaded area). 

To find an estimate of the integral error, it is convenient to start from 
the formula: 

fix) = Pi(x) + e [a,b] 

from which the approximation error is given by: 


{f{x)dx — Pi {x))dx 


< l^^l/"K)l = (5l/"K)l 


If f"{x) is bounded by M e R"*" in [a,b], then 


{f{x)dx — Pi {x))dx 


< 


j \f{x)-Pi{x)\dx<^h^ (6.15) 


The error formula 6.15 shows that large value of h = b — a causes large 
error. 


Example 6.2 In this example, the trapezoidal rule will be used to 
approximate the integral 


1 = 


1 


dx 


0 


l + x^ 
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Here, o = 0.0, b= 1.0 and f{x) = 1/(1+ a;^). The following MATLAB 
commands can be used to evaluate the integral: 

» a = 0.0; b = 1.0 ; f = @(x) l/(l+x"2) ; 

» I = (b-a)/2*(f(a)+f(b)) ; 

>> disp(I) 

0.7500 

In Python: 

In [1]: a, b, f = 0.0, 1.0, lambda x: l/(l+x**2) 

In [2]: I = (b-a)/2*(f(b)+f(a)) 

In [3]: print(I) 

0.75 


The exact value of the integral in Example 6.2 is j «0.7853981633974483. 
The approximation error is 7r/4 — 0.75 = 0.03539816339744828. 

(II) n = 2: when n = 2, h = {b— a)/2 the given data points are (o,/(o)), 
((a + 5)/2,/((a + 5)/2)) and {b,f{b)). The Lagrange interpolating poly- 
nomial is: 


P2{x) 


{x — c) {x 
(a — c)(a 


b) 


b) 


/(a)- 


i/L 

(c 


a)(x — b) 
a)(c — b) 


f(c) + 


(x — a)(x — c) 
{b — a){b — c) 


f{b), 


where c= {a+b)/2. 

Let X = a + th, 0 < t < 2 be a parametric representation of x in terms 
of t. Then, P 2 {x) can be rewritten as: 


P 2 {x) = +t(t- 2)/(c) + ^^/(5) 


Then, 

pb 


f{x)dx h f 

Jo 


(LJKLJ) /(„) + t{t - 2)/(c) + ^^/(6) 


dt 


1 ^ 4 , fa + b\ 1 


= M3/(«)+3/1 2 


b-i 


/(«)+ 4 / 


-.m 


-f{b) 


(6.16) 


The numerical quadrature (6.16) is called Simpson’s rule. 
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The leading error term in approximating j^f{x)dx by P 2 (x)dx in 
Equation (6.16) is: 


{f{x) - P 2 {x)dx 


< [ \f{x)-P 2 {x)\dx 
J a 


no 


= 0 


where a < ^ <b. Therefore, it is necessary to move to the next leading 
term of the error. Hence, 


'> pb 

{f{x)dx— / P 2 {x)dx 

J a 


< 


t(t-l)(t-2)(t-4) 

120 




= a<r]<b 


(6.17) 


Applying the Simpson’s rule to the problem of Example 6.2, with a = 
0.0,6= 1.0 and 6 = (6-o)/2 = 0.5 Using MATLAB: 


» a = 0.0 ; b = 1.0; f = Q(x) l/(l+x"2) ; 
>> h = (b-a)/2 ; 

>> c = (a+b)/2 ; 

» I_s = h/3*(f(a)+4*f(c)+f(b)) 

I_s = 

0.7833 

>> format short e 
>> dispCabs(pi/4-I_s)) 

2.0648e-03 


In Python: 

In [4]: import numpy as np 

In [5]: a, b, f = 0.0, 1.0, lambda x: l/(l+x**2) 

In [6]; c, h = (a+b)/2, (b-a)/2 
In [7]; I_s = h/3.0*(f(a)+4*f(c)+f(b)) 

In [8]: print(I_s) 

0.7833333333333333 

In [9]: print(’{0:1.4e}’.format(np.abs(np.pi/4-I_s))) 
2.0648e-03 

The errors in approximating f{x)dx by either the trapezoidal rule (6.14) 
or Simpson’s rule (6.16) become large as h= {b—a)/n gets large. To obtain 
better approximations of f{x)dx are composite versions of Newton-Cotes 
method, which partition the interval [a, 6] into A^-subintervals {N G Z“*“) by 
points xq = a < xi < ... < xn = b where Xj+i — Xj = h = (b — a)/N. Then, 
in each sub-interval [xj,Xj+i] the integral j^f{x)dx is approximated by a 
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FIGURE 6.7: The composite trapezoidal rule. 


quadrature. Two composite rules derived from the Newton-Cotes methods 
will be discussed. 

(i) The composite trapezoidal rule: the derivation of the formula starts 
from the formula: 

fb f^N r^l f^2 f^N 

/ f{x)dx= / f{x)dx= / f{x)dx+ / f{x)dx+...+ / f{x)dx, 

Ja XQ XQ Jxi — 1 

then, in each sub-interval [xj,Xj^i] the integral f{x)dx is approx- 

imated by h/2{f{xj) + f{xj+i)). This gives the formula: 

J f{x)dx « ^{f{xo) + f{xi)) + ^{f{xi) + f{x 2 )) + ... 

+ ^{fixN-l) + f{xN)) 

= ^ if{xo) + 2/(xi) + ... + 2f{xN-i) + /(a:Ar)j(6.18) 

Figure 6.7 shows the areas under trapeziums, which are in total very 
close to the area under the function’s curve. 

The following MATLAB code uses the composite trapezoidal rule with 
iV = 10x2^, A: = 0,..., 14 to approximate 1 /{1 + x‘^)dx-. 


1 %comptrapz . m 

2 ciear ; clc ; 

3 format short e 


4 a = 0.0; b = 1.0 ; f = 0 (x) l./(H-x.'-2) ; 

5 N = 10 ; Approx = [] ; 

6 fprintf ( '- \n ' ) 

7 fprintf ( ' N\t\t Approx Int.\t Error\n') ; 

8 fprintf ( ' -\n ’ ) 


9 while N < 200000 
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10 h = (b-a) /N ; 

11 X = linspace(a, b, N+1) ; 

12 I = h/2* (f {x (1) )+f (x (N+1) ) ) ; 

13 1 = 1+ h*sum (f (x (2 : N) ) ) ; 

14 Err = abs(pi/4-I) ; 

15 Approx = [Approx; [N I Err]] ; 

16 fprintf ( ' %6i\t%l . 12f\t%l . 12e\n ' , N, I, Err) ; 

17 N = 2*N ; 

18 end 

19 fprintf ( '- \n ' ) 


Running the above code, gives: 


N 

Approx Int. 

10 

0.784981497227 

20 

0.785293996739 

40 

0.785372121731 

80 

0.785391652981 

160 

0.785396535793 

320 

0.785397756496 

640 

0.785398061672 

1280 

0.785398137966 

2560 

0.785398157040 

5120 

0.785398161808 

10240 

0.785398163000 

20480 

0.785398163298 

40960 

0.785398163373 

81920 

0.785398163391 

163840 

0.785398163396 


Error 


4.166661706586083e-04 
1.041666589161050e-04 
2.604166654529561e-05 
6.510416664773366e-06 
1.627604166443142e-06 
4.069010419716079e-07 
1.017252599933016e-07 
2.543131505383656e-08 
6.357829041014895e-09 
1.589458453743475e-09 
3.973615880781267e-10 
9.934308931036640e-ll 
2.483746541770415e-ll 
6.210698622055588e-12 
1.555644502104769e-12 


In Python, the code is: 


1 # compsimp.py 

2 import numpy as np 

3 a, b, f = 0,0, 1.0, lambda x: 1/(l+x**2) 

4 ApproxTable = [] 

5 N = 10 

6 print ( '-' ) 

7 print ( ' N\t Approx Int.\t Error') 

8 print ( '-' ) 

9 while N < 200000: 

10 X, h = np.linspace(a, b, N+1), (b-a)/N 

11 I = h/2* {f {x[0] )+f (X [-1] ) ) 

12 1 = 1+ h*sum {f (x [ 1: N] ) ) 

13 Err = np. abs (np. pi/4-I) 

14 print (' {0:6.Of} format (N), ' \t ' , ... 

' {0 :1.12f} format (I), '\t', ' {0 :1. 12e} format (Err)) 
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15 N = 2*N 

16 print ( '-' ) 


Executing the code gives the results: 

runfile(’D;/PyFiles/compsimp.py’, wdir=’D;/PyFiles’) 


N 

Approx Int. 

10 

0.784981497227 

20 

0.785293996739 

40 

0.785372121731 

80 

0.785391652981 

160 

0.785396535793 

320 

0.785397756496 

640 

0.785398061672 

1280 

0.785398137966 

2560 

0.785398157040 

5120 

0.785398161808 

10240 

0.785398163000 

20480 

0.785398163298 

40960 

0.785398163373 

81920 

0.785398163391 

163840 

0.785398163396 


Error 


4.166661706586e-04 
1.041666589161e-04 
2.604166654530e-05 
6.510416664773e-06 
1.627604166443e-06 
4.069010419716e-07 
1.017252601043e-07 
2.543131505384e-08 
6.357829041015e-09 
1.589458453743e-09 
3.973615880781e-10 
9.934308931037e-ll 
2.483757644001e-ll 
6.210698622056e-12 
1.558975171179e-12 


(ii) The composite Simpson’s rule: Let N e 2Z+ ( even positive inte¬ 
ger) with N — 2k, h — (b — a)/N and Xj = a -F jh. In an interval 
[x2j,X2(j+i)], J = the composite Simpson’s rule approximales 


nb r^2k 

/ f{x)dx = / f{x)dx+...+ f{x)d: 

Ja Jxo Jx9.Jc._2 


^{f{xo)+‘if{xi) + f{x2))- 
+4:f{x2k-l) + f{X 2 k)) 


';{f{x 2 k- 2 ) 


k-1 


= ^(/(a:o) + /(a;Ar)) + y ^/(a;2j-l) + y 

i=i i=i 


^/(X2,) (6.19) 


The following MATLAB code uses the composite Simpson’s rule to com- 
pute fg l/(l + x^)dx with 7V= 10 x 2^,A: = 0,...,9 
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1 

%compsimp . m 


2 

ciear ; clc ; 


3 

format short e 


4 

a = 0.0; b = 1.0 ; f = @(x) l./{l+x.~2) ; lExact = pi/4 


5 

N = 10 ; Approx = [] ; 


6 

fprintf ( '- 

-V) 

7 

fprintf ( ' N\t\t Approx Int.\t\t Error\n') ; 


8 

fprintf ( ' - 

-\n') 

9 

while N < 10000 


10 

[I, Err] = CompSimpson(f, a, b, N, lExact) ; 


11 

Approx = [Approx; [N I Err]] ; 


12 

fprintf (' %6i\t\t%l . 12f\t\t%l . 15e\n ' , N, I, Err) ; 


13 

N = 2*N ; 


14 

end 


15 

fprintf ( ' - 

-\n') 

16 



17 

function [I, Error] = CompSimpson(f, a, b, N, lExact) 


18 

h = (b-a)/N ; 


19 

X = linspace(a, b, N+1) ; 


20 

I = h/3* (f (x(l) ) +f (X (N+1) ) ) ; 


21 

for j = 2 : N 


22 

if mod{j, 2) == 0 


23 

1 =1+ 4*h/3*f(x{j)) ; 


24 

else 


25 

1 =1+ 2*h/3*f(x{j)) ; 


26 

end 


27 

end 


28 

Error = abs (lExact-I) ; 


29 

end 



Executing the above code gives: 


N 

Approx Int. 

10 

0.785398153485 

20 

0.785398163242 

40 

0.785398163395 

80 

0.785398163397 

160 

0.785398163397 

320 

0.785398163397 

640 

0.785398163397 

1280 

0.785398163397 

2560 

0.785398163397 

5120 

0.785398163397 


Error 


9.912644483023314e-09 
1.550021222485043e-10 
2.422284595127167e-12 
3.785860513971784e-14 
3.330669073875470e-16 
1.110223024625157e-16 
3.330669073875470e-16 
2.220446049250313e-16 
1.110223024625157e-16 
1.887379141862766e-15 


In Python, the code is: 


1 #CompositeSimpson . py 

2 import numpy as np 

3 def CompSimpson(f, a, b, N, lExact): 
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4 X, h = np.linspace(a, b, N+1), (b-a)/N 

5 I = h/3,0*{f{x[0])+f (X[-1])) 

6 for j in range(l, N): 

7 if j%2 == 1: 

8 I += 4.*h/3,*f{x[j]) 

9 else: 

10 I += 2 . *h/3 , *f {x [ j ] ) 

11 Error = np. abs (lExact-I) 

12 return I, Error 

13 

14 a, b, f = 0,0, 1.0, lambda x: 1/(l+x**2) 

15 ApproxTable = [] 

16 lExact = np.pi/4 

17 N = 10 

18 print ( '-' ) 

19 print ( ' N\t Approx Int.\t Error’) 

20 print ( '-' ) 


21 while N < 10000: 

22 I, Err = CompSimpson(f, a, b, N, lExact) 

23 print (' {0: 6 .Of} format (N), '\t', ... 

'{0:1.12f} format (I), '\t', '{0:1.12e} format (Err)) 

24 N = 2*N 

25 print ( ' - ' ) 


Executing this code gives: 

runfile(’D;/PyFiles/CompositeSimpson.py’, wdir=’D;/PyFiles’) 


N 

Approx Int. 

Error 

10 

0.785398153485 

9.912644594046e-09 

20 

0.785398163242 

1.550021222485e-10 

40 

0.785398163395 

2.422284595127e-12 

80 

0.785398163397 

3.796962744218e-14 

160 

0.785398163397 

3.330669073875e-16 

320 

0.785398163397 

1.110223024625e-16 

640 

0.785398163397 

3.330669073875e-16 

1280 

0.785398163397 

2.220446049250e-16 

2560 

0.785398163397 

1.110223024625e-16 

5120 

0.785398163397 

1.887379141863e-15 


Both MATLAB and Python contain functions to compute a definite inte- 
gral f{x)dx based on trapezoidal and Simpson’s rules. The MATLAB func- 
tion trapz receives two vectors x (of discrete values in [a, 6]) and f{x) the 
corresponding values of / to x. It applies the trapezoidal rule to return the 
integral value. 

>> X = linspaceCO, 1, 101) ; 

>> f = exp(-x.~2) ; 

>> format long g 
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>> I = trapzCx, f) 

I = 

0.74681800146797 

Python also contains a trapz as a part of the scipy.integrate library. It 
differs from the MATLAB’s trapz function by receiving the vector of / before 
vector x: 

In [10]: from scipy.integrate import trapz 

In [11]: import numpy as np 

In [12]: X = np.linspaceCO, np.pi, 500) 

In [13]: g = np.zeros_like(x) 

In [14]: g[0], g[l:] = 1.0, np. sin(x[l:] )/x[l:] 

In [15] : I = trapz (g, x) 

In [16]: print(I) 

1.8519360005832526 

Also, the MATLAB’s function quad applies the Simpson’s rule to evaluate 
Ia It receives a function handle to /, a and b and then it returns the 

value of the integral: 

>> X = linspaceCO, pi, 501) ; 

>> h = @(x) sin(x)./x ; 

>> I = quad(h, 0, pi) 

I = 

1.8519e+00 

The Python’s quad function located in the scipy. integrate library 
receives same arguments as MATLAB’s quad function, but it returns two 
values: the integration value and the approximation error. 

In [17]: from scipy.integrate import quad 
In [18]: h = lambda x: np.exp(-x**2) 

In [19]: I = quad(h, 0.0, 1.0) 

In [20]: print(’Integral = ’,I[0], Error = I[l]) 

Integral = 0.7468241328124271 , Error = 8.291413475940725e-15 

6.2.2 The Gauss Integration Method 

On approximating the integral 


1= [ f{x)dx, 

J a 

a Lagrange polynomial of degree n that interpolates the integrand function 
f{x) at equally spaced points to derive a Newton-Cotes quadrature. Such a 
Newton-Cotes quadrature is expected to be exact for a polynomial of degree 
not more than n—1. 
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Unlike Newton-Cotes methods, a Gauss integration method does not pre- 
determine the positions of the interpolation points Xj^ j = 1,... ,7 t, in 
nor the corresponding weights Wj with the purpose of finding a quadrature 
that is exact for polynomials of degrees up to 2n — 1 [47]. In other words, the 
derivation of a Gauss quadrature is based on finding the set of points locations 
(Gauss points) and corresponding weights such that the quadrature is exact 
for polynomials of degrees 2n — 1 or less. 

AU the Gauss points lie in the interval [—1,1]) therefore the first step to 
start from is to use a variable transform 


X = 


b — a 
2 


s + 


b + a 
2 


-1 < s < 1 


to convert the integral problem from 


1= [ f{x)dx 

J a 


into the form 




ds. 


Through this variable transform, the point x = a is mapped into s = — 1 
and X = b to s = 1. Values a<x <b are mapped to —1 < s < 1. 

The next step is to choose a positive integer n and assuming that f{x) is 
a polynomial of degree 2n— 1. That is: 


/(x) = co + cixH- \-C 2 n-ix‘^"' \ (6.20) 


where ci,... ,C 2 n-i. 

The purpose is to find optimal sets of points a<xi < ... <Xn<b (through 
finding —1 < si < «2 < ••■ < < Sn < 1) and weights wi,W 2 ,... ,Wn such 

that 


l V /_/ (( 


b—a b+a 
( 6 . 21 ) 


is exact for f{x) = cp + CixH-|-C 2 n-ia;^"' 

Integrating Equation (6.20) from a to b gives: 

pb pb 

/ f{x)dx = / (co + CiXH - \-C 2 n-lX^^~^) dx 

J a J a 

b‘^-a? b‘^'^-a?'^ 

= co{b-a) + ci —--1-I-C 2 n -1 -r- (6.22) 

2 Zn 

From Equation (6.20): 

f{xj) =co + ciXj-|- \-C 2 n-ix^'^~^, j= l,...,n 
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Wlf{xi)-\ - \-WnfiXn) = Wi (cq H- \-C2n-ixf^ 

+W2 (co +- \-C2n—lX2^ 

H-l"U'n(coH-+ C2n— 

= Cq{wi + ... + Wn) + Ci {wiXi + ... + WnXn) 

H- \-C2n-l{w\x\ - \-WnX'^ (6.23) 


Comparing the terms of Equation (6.22) and (6.23) gives 2n nonlinear 
equations in the 2n unknowns Xj and wj, j = 1,... ,n. 


Wi+W2-\ - \-Wn 

WiXi + W2X2 H-h WnXn 

WiXi + W2X2 H-h WnXn 


b — a 
b^-a^ 
2 

b^ — 

3 


(6.24) 


WiX^ ^+W2X2 - VWnXn ^ 


' — a 
n 


The points xi,...,Xn and weights wi,..., Wn are found by solving the Sys¬ 
tem of nonlinear equations (6.24). 

Following is the derivation of the Gauss quadratures for n = 1 and n = 2: 


1. One point Gauss quadrature: In the case that n = 1, there is one point 
xi and one weight wi to be found. Since 2n — 1 = 1, it is assumed that 
f{x) = co + cix. The equations in xi and wi are: 


wi = b — a 


b^-a^ 


WlXl = 


Solving this System of equations gives: 


wi = b — a and xi = 


b + a 


The Gauss quadrature is: 

rb 


f f{x)dx = {b-a)f 
J a 


b + a 


(6.25) 


2. Two points Gauss quadrature: For n = 2, there are two points a:i,a ;2 
and weights wi,W 2 to be found. Since 2n — 1 = 3, the function f{x) is 
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assumed to be cubic {i.e. f{x) = cq + cix + C 2 X^ + c^x^). The System of 
equations in xi,X 2 ,wi and W 2 is: 


W1 + W2 
W1X1 + W2X2 

WiXi + W2X2 

Q Q 

WiXi + W2X2 


= b — a 
b^-a^ 
2 

_ b^ -a? 

3 

b^-a^ 

4 


The following Python code can by used to find the solution of the nonlinear 
System: 

In [21]: import sympy as smp 

In [22]: from sympy.solvers import solve 

In [23]: wl, w2, xl, x2, a, b = smp.symbois(’wl, s2, xl, x2, 
a, b’, cls=smp.Symbol) 

In [24]: Syst = [wl+w2-(b-a), wl*xl+w2*x2-(b**2-a**2)/2, \ 

...: wl*xl**2+w2*x2**2-(b**3-a**3)/3, 
wl*xl**3+w2*x2**3-(b**4-a**4)/4] 

In [25]: vrs = [wl, w2, xl, x2] 

In [26]: Sol = solve(Syst, vrs) 

In [27]: print(Sol[0]) 

(-(a - b)/2, -(a - b)/2, -sqrt(3)*a/6 + a/2 
+ sqrt(3)*b/6 + b/2, a/2 + b/2 + sqrt(3)*(a - b)/6) 

In [28]: print(Sol[1]) 

(-(a - b)/2, -(a - b)/2, sqrt(3)*a/6 + a/2 
- sqrt(3)*b/6 + b/2, a/2 + b/2 - sqrt(3)*(a - b)/6) 


The Python symbolic library gives two Solutions of the nonlinear system, 
in which the values of xi and X 2 exchange. If we put a condition xi < X 2 
the solution of the nonlinear system is: 


b—a 

wi = W2= > 2^1 


b + a 
2 


b—a 


■\/3 and X 2 


b + a 
2 


b — a 


73 


The two-points Gauss quadrature is: 

rb 


(6.26) 

Example 6.3 In this example, the one- and two-points Gauss quadratures 
will be used to evaluate the integral: 


xdx 
l + 2x^ 
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The interval [0,2] will be divided into N sub-intervals {N is a positive integer) 
by points 0 = a;o <a;i < ■■■ <xn = 2. Each sub-interval has a length h = 2/N. 
Then, in each subinterval [xj,Xj-i^i] the one- or two points Gauss quadrature 
will be used to approximate f{x)dx. 

The 1-point Gauss quadrature approximates f{x)dx as 


Then, 


N-l 


r^j+1 


N-l 


/(a;)dx« ^ / f{x)dx = h^ f 

3=0 3=0 


Xi + 






The 2-points Gauss quadrature approximates f{x)dx as 


Then, 


r^3+i h 

f{x)dxfv- ( / 

+/ 


Xj-\-Xj-^\ Xj-^\ — Xj 


6 


V3 


Xj Xj-^\ 

2 6 


73 


f{x)dx 


N-l 

E 

3=0 •'^3 
, N-l 


rj + 1 


= i T. f 


3=0 


f{x)dx 


Xj Xj-^"]^ Xj-^\ 


6 


73 


+/ 


Xj ~\~ Xj-^\ Xj-i^\ 

2 6 


73 


The following MATLAB code shows the approximations and errors of the 
1- and 2-points Gauss quadratures for different values of 7V(= 10 x 2^,fc = 
0 ,..., 10 ). 


1 ciear ; clc ; 

2 f = @(x) x./{H-2*x."2) ; 

3 a=0.0;b=2.0; 

4 N = 10 ; 

5 lExact = log{9) /4 ; 

6 GLlApprox = [] ; GL2pApprox = [] ; 

7 

8 while N < 20000 

9 x = linspace(a, b, N+1) ; 

10 h = (b-a) /N ; 

II = h*sum{f ((x(1:N)+x(2 :N+1) )/2)) ; 


11 
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12 Errl = abs (Il-IExact) ; 

13 GLlApprox = [GLlApprox; [N II Errl]] ; 

14 12 =.. . 

h/2*sum(f({x{1:N)+x(2:N+1))/2-(x(2;N+1)-x(1:N)) *sqrt (3)/6) 

15 +. . .f( (x{1:N)+x(2:N+1) )/2+{x(2:N+1)-x{1:N)) *sqrt (3)/6)) ; 

16 Err2 = abs {l2-IExact) ; 

17 GL2pApprox = [GL2pApprox; [N 12 Err2]] ; 

18 N = 2*N ; 

19 end 

20 

21 fprintf (' Integration with 1-point Gauss quadrature\n ' ) 

22 fprintf ( ' - 

23 fprintf ( ' N\t\t Approx Int.\t\t Error\n') ; 

24 fprintf ( '- 

25 [m, L] = size (GLlApprox) ; 

26 for j = 1 : m 

27 fprintf('%6i\t\t%1.12f\t\t%1.15e\n';. GLlApprox ( j , 1) , 

28 GLlApprox{ j ,2), GLlApprox{ j ,3)) ; 

29 end 

30 fprintf ( ' - 

31 

32 fprintf (' Integration with 2-points Gauss quadrature\n ' ) 

33 fprintf ( ’ -\n ’ ) 

34 fprintf ( ' N\t\t Approx Int.\t\t Error\n') ; 

35 fprintf ( '- \n ' ) 

36 [m, L] = size (GL2pApprox) ; 

37 for j = 1 : m 

38 fprintf('%6i\t\t%1.12f\t\t%1.15e\n', GL2pApprox(j,1), ... 

39 GL2pApprox{j,2), GL2pApprox{j,3)) ; 

40 end 

41 fprintf ( ' - \n\n\n ' ) 


By executing the code, the following results are obtained: 


Integration with 1-point Gauss quadrature 


N 

Approx Int. 

Error 

10 

0.551141201821 

1.835057487226677e-03 

20 

0.549760289497 

4.541451628828908e-04 

40 

0.549419404209 

1.132598753686986e-04 

80 

0.549334442201 

2.829786668079315e-05 

160 

0.549313217734 

7.073400441592881e-06 

320 

0.549307912618 

1.768283512948443e-06 

640 

0.549306586401 

4.420667167881476e-07 

1280 

0.549306254850 

1.105164195713826e-07 

2560 

0.549306171963 

2.762908835052258e-08 

5120 

0.549306151241 

6.907272531719855e-09 

10240 

0.549306146061 

1.726818577019174e-09 


■\n') 

■\n') 


■\n\n\n' ) 
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N 

Approx Int. 

10 

0.549301461604 

20 

0.549305863669 

40 

0.549306126963 

80 

0.549306143251 

160 

0.549306144266 

320 

0.549306144330 

640 

0.549306144334 

1280 

0.549306144334 

2560 

0.549306144334 

5120 

0.549306144334 

10240 

0.549306144334 


Error 


4.682729895844062e-06 
2.806649841424758e-07 
1.737151811287419e-08 
1.083119705036495e-09 
6.765477067460779e-ll 
4.227507233167671e-12 
2.640110352558622e-13 
1.576516694967722e-14 
1.554312234475219e-15 
6.661338147750939e-16 
4.440892098500626e-16 


The Python code is: 


1 import numpy as np 

2 f = lambda x: x/(1.0+2,0*x**2) 

3 a, b= 0.0, 2.0 

4 N = 10 

5 lExact = np.log(9)/4 

6 GLlApprox, GL2Approx = [], [] 

7 while N < 20000: 

8 11 , 12 = 0 . 0 , 0.0 

9 x = np.linspace(a, b, N+1) 

10 h = (b-a) /N 

11 for j in range (N) : 

12 II += h* (f ( (x[ j]+x[ j + 1] )/2.0) ) 

13 12 += . . . 

h/2*(f((x[j]+x[j + l])/2-(x[j + l]-x[j]) *np . sqrt (3 . ) / 6 . ) +\ 

14 f((x[j]+x[j + l])/2+(x[j + l]-x[j]) *np. sqrt (3 . ) / 6 . ) ) 

15 Errl = abs (Il-IExact) ; 

16 GLlApprox.append([N, II, Errl]) 

17 Err2 = abs {l2-IExact) 

18 GL2Approx.append([N, 12, Err2]) 


19 N = 2*N 

20 

21 print (' Integration with 1-point Gauss quadrature\n ' ) 

22 print ( ’- \n ' ) 

23 print ( ' N\t\t Approx Int.\t\t Error\n') ; 

24 print ( ' -\n ’ ) 

25 m = len (GLlApprox) 

26 for j in range (m): 

27 print ('{0:6. Of }’ . format (GLlApprox[j] [0]), ' \t ' , \ 

28 '{0:1.12f}’. format (GLlApprox[j][l]), '\t', \ 

29 '{0:1.12e}’. format (GLlApprox[j][2])) 

30 print ( ' - \n\n\n') 

31 

32 print (' Integration with 2-points Gauss quadrature\n ' ) 
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33 print ( '- \n ' ) 

34 print ( ' N\t\t Approx Int.\t\t Error\n') ; 

35 print ( '- \n ' ) 

36 m = len (GL2Approx) ; 

37 for j in range (m) : 

38 print ('{0:6. Of }' . format {GL2Approx [j] [0] ), ' \t ' , \ 

39 '{0:1, 12f}’. format {GL2Approx[j][l]), '\t'/ \ 

40 '{0:1. 12e}'. format {GL2Approx[j][2])) 

41 print ( ' - \n\n\n ' ) 


Since the number of subintervals is doubled in the above example, the rates 
of convergences for the 1- and 2-points Gauss quadratures can be computed 
using the rule: 

/ ErrorM \ 

The following Python code can be used to see the rates of convergences of the 
two methods: 


1 

RConvl = [] 


2 

print ('Rates of convergence for 1-point Gauss quadrature: 

\n') 

3 

print ( '- \n ’ ) 


4 

print ( ' N\t Conv. Rate\n') 


5 

print ( ' -\n ' ) 


6 

for k in range (m-1): 


7 

RConvl.append(np.log2(GLlApprox [k] [2]/GLlApprox[k+1 ][2 

] ) ) 

8 

print ('{0:4. Of }' . format (GLlApprox [k] [0]), ' \t ... 

{0: 2 . 3f }' . format (RConvl[-1])) 


9 

10 

print ( '- \n ’ ) 


11 

RConv2 = [] 


12 

print ('Rates of convergence for 2-points Gauss quadrature: 

\n') 

13 

print ( ’- \n ’ ) 


14 

print ( ' N\t Conv. Rate\n') 


15 

print ( ' -\n ' ) 


16 

for k in range (m-1): 


17 

RConv2.append(np.log2(GL2Approx [k] [2]/GL2Approx[k+1][2 

] ) ) 

18 

print ('{0:4. Of }' . format (GL2Approx [k] [0]), ' \t ... 

{0:2. 3f }' . format (RConv2[-1])) 


19 

print ( '- \n ’ ) 



Executing the code gives: 


Rates of convergence for 1-point Gauss quadrature; 


N Conv. Rate 


10 2.015 
20 2.004 
40 2.001 
80 2.000 
160 2.000 
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320 2.000 

640 2.000 

1280 2.000 

2560 2.000 

5120 2.000 


Rates of convergence for 2-points Gauss quadrature: 


N Conv. Rate 


10 

4.060 

20 

4.014 

40 

4.003 

80 

4.001 

160 

4.000 

320 

3.998 

640 

4.010 

1280 

4.402 

2560 

0.000 

5120 

-0.652 


The above results show that the 1-point Gauss quadrature behaves as a 
second order method and the 2-point Gauss quadrature behaves as a fourth- 
order method. 

Higher-order Gauss quadrature methods can be obtained for n > 3. In 
fact, in n-points Gauss quadrature method, the points xi,X 2 , ■ ■ ■ ,Xn are the 
roots of the Legendre polynomial Pn{x). Hence, this class of points is referred 
to as Legendre-Gauss points. The Python function legendre located in 
scipy. special library receives a parameter n and returns the coefhcients 
of the corresponding n*^-degree Legendre polynomial [5, 44]. 

In [29]: from scipy.special import legendre 
In [30]: legendre(4) 

Out [30]: polyld([ 4.37500000e+00, 4.85722573e-16, 

-3.75000000e+00, 2.42861287e-16, 3.75000000e-01] ) 

In [31]: print(legendre(4)) 

4 3 2 

4.375 X + 4.857e-16 x - 3.75 x + 2.429e-16 x + 0.375 
In [32]: np.roots(legendre(4)) 

0ut[32]: array([ 0.86113631, -0.86113631, 0.33998104, 

-0.33998104]) 

The last line np. roots (legendre (4)) gives the point locations of the 4-points 
Legendre-Gauss quadrature. 
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FIGURE 6.8: The positions of Legendre-Gauss points for polynomials of 
degrees n= 


Figure 6.8 shows the distributiori of Legendre-Gauss points for polynomials 
of degrees 1 to 10. 

Having the n coordinates of the Legendre-Gauss points xi,.. .,Xn, the cor- 
responding weights wi,..., Wn can be found by solving the linear system: 


1 

1 

1 

1 


Wi 


2 

Xi 

X2 


Xfi 


W2 


0 

^2 

^2 

^2 





2 

Xi 



X-n 


W3 

= 

3 


xr' 

xr' 

1 

. . 1 




l-(-l)" 

n 


The matrix at the left-hand side of Equation (6.27) is the transpose matrix 
of the Vandermonde matrix of type nxn, constructed from the vector of points 

, . . . , Xfi. 

The following Python code (FindLGParams .py) includes a function LGpw 
that receives a parameter n and returns the corresponding Legendre-Gauss 
points and weights. 


1 #FindLGParams .py 

2 

3 from scipy.special import legendre 

4 import numpy as np 

5 

6 def LGpw(n): 

7 s = list (np.sort(np,roots(legendre(n)))) 

8 X = (np.fliplr(np.vander(s))).T 

9 Y = np.array([(1-(-1)**j)/j for j in range(l, n+1)]) 

10 w = np.linalg.solve(X, Y) 

return s, w 


11 
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12 


13 s, w = LGpw (3 ) 

14 print('n = 3:\n') 

15 print ( ' Points are: s, '\n Weights are:', w) 

16 

17 print (' n = 6:\n') 

18 s, w = LGpw ( 6) 

19 print (' Points are: s, ' \n Weights are:', w) 


By executing the code the following results are obtained: 


runfile(’D;/PyFiles/FindLGParams.py’, wdir=’D;/PyFiles’) 
n = 3; 

Points are; [-0.7745966692414835, 0.0, 0.7745966692414834] 
Weights are; [0.55555556 0.88888889 0.55555556] 

n = 6; 

Points are; [-0.9324695142031514, -0.6612093864662644, 
-0.23861918608319704, 0.23861918608319688, 0.6612093864662634, 
0.9324695142031519] 

Weights are; [0.17132449 0.36076157 0.46791393 0.46791393 
0.36076157 0.17132449] 

Example 6.4 This example uses the function LGpw to find the points and 
weights of a 5-points Legendre-Gauss quadrature, then applies it to find the 
integration in Example 6.3; 



The Python code LG5Approx.py implements the method; 


1 from scipy.special import legendre 

2 import numpy as np 

3 

4 def LGpw(n): 

5 s = list (np.sort (np.roots(legendre(n)))) 

6 X = (np.fliplr(np.vander(s))).T 

7 Y = np.array([ (1-(-1)**j)/j for j in range(l, n+1)]) 

8 w = np.linalg.solve(X, Y) 

9 return s, w 
10 

11 s, w = GLpw (5 ) 

12 

13 a, b = 0.0, 2.0 

14 f = lambda x: x/(1.0+2,0*x**2) 

15 N = 5 

16 lExact = np . log ( 9.0)/4 . 0 

17 GLSApprox = [] 

18 while N < 1000: 


x, h = np.linspace(a, b, N+1), (b-a)/N 


19 
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20 I = 0.0 

21 for j in range (N): 

22 xm = {x [ j ]+x [ j + 1 ] )/2.0 

23 I += h/2. 0*sum ([w[k]*f{xm+h/2+s[k]) for k in ... 

range ( len (s))]) 

24 Err = abs (I-IExact) 

25 GL5Approx.append([N, I, Err]) 


26 N = 2*N 

27 

28 print (' Integration with 5-point Gauss quadrature\n ' ) 

29 print ( ’- \n ' ) 

30 print ( ’ N\t\t Approx Int.\t\t Error\n') ; 

31 print ( ’ -\n ’ ) 

32 m = len (GL5Approx) 

33 for j in range (m): 

34 print ('{0:6. Of }’ . format (GLSApprox[j] [0]), ' \t ' , \ 

35 '{0:1. 12f}’. format (GLSApprox[j][l]), '\t'/ \ 

36 '{0:1. 12e}'. format (GLSApprox[j][2])) 

37 print ( ' - \n\n\n') 


Executing the Python code gives the following results: 

runfile(^D;/PyFiles/LGSApprox.py^, wdir= ;/PyFiles ^) 
Integration with 5-point Gauss quadrature 


N 

Approx Int. 

Error 

5 

0.549306144553503 

3.995001813766e-10 

10 

0.549306144334415 

6.564653283378e-13 

20 

0.549306144334055 

6.063411283909e-16 

40 

0.549306144334055 

2.021137094636e-16 

80 

0.549306144334055 

2.021137094636e-16 

160 

0.549306144334055 

4.042274189272e-16 

320 

0.549306144334055 

4.042274189272e-16 

640 

0.549306144334054 

1.4147959662450-15 


The MATLAB code LGApprox is: 


1 n = s ; 

2 [s, w] = LGpw(n) ; 

3 a = 0.0 ; b = 2.0 ; f = @(x) x./(H-2*x."2) ; 

4 N = S ; 

5 lExact = log(9.0)/4 ; 

6 LGSApprox = [] ; 

7 while N < 1000 

8 X = linspace(a, b, N+1) ; 

9 h = (b-a)/N ; 

10 I = 0,0 ; 

11 for j = 1 : N 

= (x (j)+x(j + 1) ) /2 ; 


12 


xm 
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13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 
63 


1 =1+ h/2. 0*sum (w.*f(xm+h/2*s)) ; 

end 

Err = abs(I - lExact) ; 

LGSApprox = [LGSApprox; [N, I, Err]] ; 

N = 2*N ; 

end 

fprintf (' Integration with 5-point Gauss quadrature\n ' ) 


fprintf ( ' -\n ' ) 

fprintf ( ' N\t\t Approx Int.\t\t Error\n') ; 

fprintf ( '- \n ' ) 


m = length (LGSApprox) ; 
for j = 1 : m 

fprintf ( ' %4i\t%l . 15f\t%l . 12e\n ' , LG5Approx(j, 1), . . . 

LG5Approx(j, 2 ), LG5Approx(j, 3)/IExact) ; 


end 

fprintf ( '- \n ' ) 

function [s, w] = LGpw(n) 


s = sort(roots (legendreV(n))) ; 

X = fliplr(vander (s))' ; 

Y = zeros (n, 1) ; 

for j = 1 : n 

Y(j) = {l-{-l)''j)/j ; 

end 

w = X\Y ; 

end 

function Legn = legendreV(n) 

%legendreV . m 

% This function receives a parameter n representing the ... 
polynomial degree 

% and returns the cofficients of Legendre polynomial of ... 

degree n using the 
% recursive relation 
% P_n (x) = ... 

{ (2n-l) /n) xP_{n-l} (X) -( (n-1) /n) P_{n-2} (x) 

% Written by Eihab B.M. Bashier, ebashier@du . edu . om 

if n == 0 

Legn = [0, 1] ; 

elseif n == 1 

Legn = [1, 0] ; 

else 

L1 = conv{[l, 0], legendreV(n-1)) ; 

L2 = legendreV(n-2) ; 

if length{L2) < length{Ll) 

L2 = [zeros(l;. length (L1) -length {L2 ) ) L2] ; 

end 

Legn = {2*n-l)/n*Ll-{n-1)/n+L2 ; 

end 


64 


end 
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FIGURE 6.9: The distributiori of Legendre-Gauss-Lobbatto points in the 
interval [—1,1] for n = 2,3,..., 11. 


The numerical quadratures based on Legendre-Gauss points are open. That 
is they are not including the boundary points —1.0 and 1.0 (hence the inte¬ 
gration boundaries a and b). There is another class of Gauss quadratures that 
imposes the choice of the boundary points —1.0 and 1.0. So, in addition to 
xi = —1.0 and ccn = 1.0, an n-points closed Gauss quadrature may choose 
the roots of {1 — x‘^)P^_i{x) where P^_i{x) is the derivative of the Legen- 
dre polynomial of degree n—1 with n>2. This kind of points is referred as 
Legendre-Gauss-Lobbatto (LGL) points. 

Figure 6.9 shows the Legendre-Gauss-Lobbatto points for different values 
of n. 

The corresponding weights to the n-LGL points can be obtained from 
Equation (6.27). For n = 2 and n = 3, the resulting quadratures coincide with 
the trapezoidal and Simpson’s rules. For n. > 4 the resulting quadratures are 
different from those obtained by Newton-Gotes methods. 

Because the boundary points are enforced to be included among the n 
LGL-points, the accuracy of the method is reduced by 2. Hence, instead of 
being accurate up to degree 2n — 1 they are accurate up to degree 2n —3. 

The Python code LGLParams includes a function LGLpw that receives a 
positive integer n>2 and returns the corresponding n-LGL points —1 = xi < 
X 2 < ■ ■ ■ < Xn = ^ and weights wi,... ,Wn. It then approximates the definite 
integral 



using 4-LGL points. 


l + 2x^’ 






Numerical Integration 


157 


1 

import numpy as np 


2 

from scipy . special import legendre 


3 

def LGLpw(n): 


4 

s = list (np . sort(np . roots([-1 , 0, ... 

1]*np.polyder(legendre(n-1) ) ) ) ) 


5 

X = (np . fliplr(np . vander(s))) . T 


6 

Y = np . array([(1-(-1)**j)/j for j in range(l, n+1)]) 


7 

w = np , linalg . solve(X, Y) 


8 

return s, w 


10 

n = 4 


11 

s, w = LGLpw(n) 


13 

a, b = 0.0, 2.0 


14 

f = lambda x: x/(1 . 0+2 , 0*x**2) 


15 

N = 5 


16 

lExact = np . log(9 . 0)/4 . 0 


17 

LGLSApprox = [] 


18 

while N < 1000: 


19 

X, h = np . linspace(a, b, N+1), (b-a)/N 


20 

I = 0.0 


21 

for j in range (N): 


22 

xm = (x[j]+x[j + l])/2.0 


23 

I += h/2 . 0*sum ([w[k]*f{xm+h/2+s[k]) for k in ... 
range ( len (s))]) 


24 

Err = abs (I-IExact) 


25 

LGLSApprox . append([N, I, Err]) 


26 

27 

N = 2*N 


28 

print (’ Integration with 5-point Gauss quadrature\n' ) 


29 

print ( '- 

-\n') 

30 

print ( ' N\t\t Approx Int.\t\t Error\n') ; 


31 

print ( '- 

-\n') 

32 

m = len (LGLSApprox) 


33 

for j in range (m): 


34 

print ('{0:6.Of}' . format (LGLSApprox[j] [0]) , ' \t' , \ 


35 

'{0:1.15f}' . format (LGLSApprox [j][l]), '\t\ \ 


36 

'{0:1.12e}' . format (LGLSApprox[j] [2]/lExact) ) 


37 

38 

print ( '- 

-\n\n\n' ) 

39 

RConvS = [] 


40 

print ('Rates of convergence for ' +str (n)+ '-points ... 
Legendre-Gauss-Lobbato quadrature: \n') 


41 

print ( '-\n ' ) 


42 

print ( ' N\t Conv. Rate\n') 


43 

print ( '-\n ’ ) 


44 

for k in range (m-1): 


45 

RConvS , append(np . log2(LGLSApprox[k][2]/LGLSApprox[k+1] 

[2] ) ) 

46 

print ('{0:4.Of}' . format (LGLSApprox[k] [0]), '\t ... 

{0:2.3f}' . format (RConvS[-1])) 


47 

print ( '-\n ' ) 



Executing the code gives the following results: 


runfile(’D:/PyFiles/LGLParams.py’, wdir=’D:/PyFiles’) 
Integration with 5-point Gauss quadrature 
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N 

Approx Int. 

Error 

5 

0.549303859258719 

4.159930413361e-06 

10 

0.549306121271021 

4.198575552244e-08 

20 

0.549306144007535 

5.944232913986e-10 

40 

0.549306144329063 

9.088447173451e-12 

80 

0.549306144333977 

1.410753692056e-13 

160 

0.549306144334054 

2.021137094636e-15 

320 

0.549306144334055 

4.042274189272e-16 

640 

0.549306144334054 

1.212682256782e-15 


Rates of convergence for 4-points Legendre-Gauss-Lobbato 
quadrature; 


N 

Conv. Rate 

5 

6.631 

10 

6.142 

20 

6.031 

40 

6.009 

80 

6.125 

160 

2.322 

320 

-1.585 


The table of rates of convergence, shows that the Gauss quadrature based 
on 4-LGL points behaves as order 6 integration method. 

The MATLAB code is: 


1 n = 4 ; 

2 [s, w] = LGLpw(n) ; 

3 a = 0.0 ; b = 2.0 ; f = @{x) x./(H-2*x."2) ; 

4 N = 5 ; 

5 lExact = log{9.0)/4 ; 

6 LGL4Approx = [] ; 

7 while N < 1000 

8 X = linspace(a, b, N+1) ; 

9 h = (b-a)/N ; 

10 I = 0,0 ; 

11 for j = 1 : N 

12 xm = {x { j )+x ( j + 1) )/2 ; 

13 1=1+ h/2. 0*sum {w.*f{xm+h/2*s)) ; 

14 end 

15 Err = abs(I - lExact) ; 

16 LGL4Approx = [LGL4Approx; [N, I, Err]] ; 

17 N = 2*N ; 

18 end 
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20 fprintf (’ Integration with 4-point Gauss quadrature\n' ) 

21 fprintf ( ’-\n ' ) 

22 fprintf ( ' N\t\t Approx Int.\t\t Error\n') ; 

23 fprintf ( '-\n ' ) 

24 m = length (LGL4Approx) ; 

25 for j = 1 : m 

26 fprintf (' %4i\t%l.15f\t%l.12e\n' , LGL4Approx(j, 1), ... 

27 LGL4Approx{j, 2 ), LGL4Approx(j, 3)/IExact) ; 

28 end 

29 fprintf ( '-\n ' ) 

30 

31 RConv4 = [] ; 

32 fprintf (' Rates of convergence for 4-points Gauss quadrature: \n') 

33 fprintf ( '-\n ' ) 

34 fprintf ( ' N\t Conv. Rate\n') 

35 fprintf ( '-\n ' ) 

36 for k = 1 :m-l 

37 RConv4 =[RConv4; (log2 {LGL4Approx(k, 3)/LGL4Approx{k+1, ... 

3)))] ; 

38 fprintf (' %4i\t%l.12e\n' , LGL4Approx{k, 1), RConv4(end)) ; 

39 end 

40 fprintf ( '-\n ’ ) 

41 

42 function [s, w] = LGLpw(n) 

43 s = sort (roots(conv ([-1, 0, 1], polyder (legendreV(n-1))))) ; 

44 X = fliplr (vander (s) ) ' ; 

45 Y = zeros (n, 1) ; 

46 for j = 1 : n 

47 Y(j) = {l-{-l)''j)/j ; 

48 end 

49 w = X\Y ; 

50 end 

51 

52 function Legn = legendreV(n) 

53 if n == 0 

54 Legn = [0, 1] ; 

55 elseif n == 1 

56 Legn = [1, 0] ; 

57 else 

58 L1 = conv([l, 0], legendreV(n-1)) ; 

59 L2 = legendreV(n-2) ; 

60 if length (L2) < length (Ll) 

61 L2 = [zeros(l;. length (Ll) -length (L2) ) L2] ; 

62 end 

63 Legn = (2*n-l)/n*Ll-(n-1)/n+L2 ; 

64 end 

65 end 

A quadrature based on n Legendre-Gauss points is accurate for polynomi- 
als of degree 2n — 1 or less. Imposing the inclusion of the two boundary points 
as Legendre-Gauss-Lobbato points reduces the accuracy by 2 to 2n — 3. 

Another class of Gauss points is the Legendre-Gauss-Radau (LGR) points 
in which one boundary point is selected among the n points. The LGR-points 
are the roots of Pn-i{x) + Pn{x), where Pn{x) is the Legendre polynomial 
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FIGURE 6.10: The distributiori of Legendre-Gauss-Radau points in the inter- 
val [—1,1] for n = 1,3,, 11. 


of degree n. The positions of the n LGR-points are shown in Figure 6.10 for 
n = 1,..., 11. 

A MATLAB function PolyAdd the receives vectors of coefhcients of two 
polynomials Pi of degree n and ^ 2 ( 21 ) of degree m and returns the vectors of 
coefhcients of Pi{x) + P 2 {x) is described by the following code: 


1 

function P 

= PolyAdd(Pl, P2) 

2 

nl = length(Pl) ; n2 = length(P2) ; 

3 

if nl = 

= n2 

4 

P = 

P1 + P2 ; 

5 

elseif 

nl > n2 

6 

P = 

Pl + [zeros (1, nl-n2) P2] ; 

7 

else 


8 

P = 

[zeros(l, n2-nl) Pl]+P2 ; 

9 

end 


10 

end 



The MATLAB function LGRpw receives a positive integer n and uses the 
function PolyAdd to return the LGR points and weights. 


1 function [s, w] = LGRpw(n) 

2 s = sort (roots (poly.add(legendreV(n-1), legendreV(n)))) ; 

3 X=fliplr(vander (s))' ; 

4 Y = zeros (n, 1) ; 

5 for j = 1 : n 

6 Y(j) = {l-{-l)''j)/j ; 

7 end 

8 w = X\Y ; 

9 end 
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Running the MATLAB code: 


1 

% LGRApp.m 


2 

ciear ; clc ; 


3 

n = 3 ; 


4 

[s, w] = LGRpw(n) ; 


5 

a = 0.0 ; b = 2.0 ; f = @(x) x./(l+2*x."2) ; 


6 

N = 5 ; 


7 

lExact = log(9.0)/4 ; 


8 

LGRApprox = [] ; 


9 

while N < 1000 


10 

X = linspace(a, b, N+1) ; 


11 

h = (b-a) /N ; 


12 

I = 0.0 ; 


13 

for j = 1 : N 


14 

xm = (x(j)+x(j+1))/2 ; 


15 

1 =1+ h/2. 0*sum (w.*f{xm+h/2*s)) ; 


16 

end 


17 

Err = abs(I - lExact) ; 


18 

if Err < eps 


19 

break ; 


20 

end 


21 

LGRApprox = [LGRApprox; [N, I, Err]] ; 


22 

N = 2*N ; 


23 

24 

end 


25 

fprintf ( ’ %s%i%s\n ' , 'Integration with Gauss quadrature based 
on', n, '-LGR points:') 


26 

fprintf ( '- 

-\n') 

27 

fprintf ( ' N\t\t Approx Int.\t\t Error\n') ; 


28 

fprintf ( '- 

-\n') 

29 

m = length (LGRApprox) ; 


30 

for j = 1 : m 


31 

fprintf (' %4i\t%l.15f\t%l.12e\n' , LGRApprox(j, 1), ... 


32 

LGRApprox(j, 2 ), LGRApprox(j, 3)/lExact) ; 


33 

end 


34 

35 

fprintf ( ' - 

-\n') 

36 

RConv4 = [] ; 


37 

fprintf (' %s%i%s\n ' , 'Rates of convergence for Gauss ... 
quadrature based on ', n, '-LGR points') 


38 

fprintf ( '- \n ' ) 


39 

fprintf ( ' N\t Conv. Rate\n') 


40 

fprintf ( '- \n ' ) 


41 

for k = 1 :m-l 


42 

RConv4 =[RConv4; (log2 {LGRApprox(k, 3 ) /LGRApprox(k+1, 3))) 

; 

43 

fprintf (' %4i\t%3.3f\n ' LGRApprox (k, 1), RConv4 (end) ) ; 


44 

end 


45 

fprintf ( '- \n ' ) 
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gives the results: 

Integration with Gauss quadrature based on 3-LGR points; 


N 

Approx Int. 

Error 

5 

0.549307676087172 

2.788523546236e-06 

10 

0.549306137628403 

1.220749463804e-08 

20 

0.549306144241099 

1.692233412952e-10 

40 

0.549306144332712 

2.444363202253e-12 

80 

0.549306144334036 

3.375298948042e-14 

160 

0.549306144334055 

4.042274189272e-16 

320 

0.549306144334055 

4.042274189272e-16 

640 

0.549306144334054 

1.414795966245e-15 


Rates of 

convergence for Gauss quadrature based on 3-LGR points 

N 

Conv. Rate 

5 

7.836 

10 

6.173 

20 

6.113 

40 

6.178 

80 

6.384 

160 

0.000 

320 

-1.807 


The Python function LGRpw(n) receives an integer n and returns the corre- 
sponding n-LGR points and weights: 


1 from scipy.special import legendre 

2 import numpy as np 

3 def LGRpw(n): 

4 s = list (np.sort(np.roots(legendre(n-1)+legendre(n)))) 

5 X = (np.fliplr(np.vander(s))).T 

6 Y = np.array([(1-(-1 )**j)/j for j in range(l, n+1)]) 

7 w = np.linalg.solve(X, Y) 

8 return s, w 

Applying to the integral 


1 = 


-dx. 


0 


1 + 2x2 
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with n = 4 gives the following results: 

Integration with Gauss quadrature based on 4-LGR points 
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N 

Approx Int. 

Error 

5 

0.549306013521088 

2.381421874512e-07 

10 

0.549306144353480 

3.536282517630e-ll 

20 

0.549306144334126 

1.291506603473e-13 

40 

0.549306144334055 

4.042274189272e-16 

80 

0.549306144334055 

2.021137094636e-16 

Rates 

of convergence for Gauss 

quadrature based on 4-LGR points 


N 

Conv. Rate 

5 

12.717 

10 

8.097 

20 

8.320 

40 

1.000 
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Solving Systems of Nonlinear Ordinary 
Differential Equations 


Abstract 

Differential equations have wide applications in modelling real phenomena 
[61, 62]. Analytical Solutions of differential equations can be found for few 
special and simple cases. Hence, numerical methods are more appropriate in 
finding Solutions of such differential equations. 

This chapter discusses some of the numerical methods for solving a System 
of initial value problems: 

^ = f{x,y), y{a) = yo, (7.1) 

where y : R —>■ R*^, / : R x R"^ —> R*^ and x G [a,b]. Here, we assume that the 
functions fj{x,y) are Lipschitz continuous in [a,6] for all j = l,...,n and at 
least One function fkix,y) is nonlinear for some k G {l,2,...,n}. 

It is divided into five sections. The first, second and third sections discuss 
the general idea of Runge-Kutta methods, explicit and implicit Runge-Kutta 
methods. The fourth section discusses the MATLAB® built-in functions for 
solving Systems of differential equations. The fifth section discusses the scipy 
functions and gekko Python methods for solving initial value problems. 


7.1 Runge-Kutta Methods 

An M-stage Runge-Kutta method for solving Equation (7.1), is any numerical 
method characterized by a triplet A = {a)ij G ,b — {b)i G and c = 

{c)i G R^ , i,j = 1,...,AI [13]. The triplet {A,b,c) satisfies the Butcher array 



Ci 

Oli 

aiM 

A 





cm 

OmI • 

■ a,MM 



bi . 

bM 
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The parameters Cj, j = are such that 0 < ci < C 2 < ... < cm < 1 • 

The parameters aij, i,j = satisfy 

M 

^ ^ ^ij — Q, 

J = 1 

and the parameters bi, i = 1,... ,M satisfy 

M 

Y.h = i 

i=l 

A M-stage Runge-Kutta method, for solving (7.1) is described below. 
Divide the time interval [a,6] into N sub-intervals [xi,Xi^i],i = 0,...,N, each 
is of length hi = Xj+i —Xi. Then, a = xq < xi < ■ ■ ■ < xn = b. 

Given y{xi), we compute y{xi-^-l), by integrating equation (7.1) on 
[xi,xi+i], as 

r^i+i 

y{xi+i)=y{xi)+ f{x,y{x))dx, (7.2) 

JXi 

On each sub-interval [xi,Xi-^-i] let = Xi + Cjhi for j = The 

slope is then evaluated as f{x\^\y{x\^'^)), and the weighted sum of the slope 
over the sub-interval j is given by 

M 

/(xpPy(a;pp) «^ajz/(a:fPy(a;fp) 

1=1 

where Y^iLiO-ji = O- Then 

y(a:pp = y{x['^'>) +J f{x,y{x))dx 

M 

~ y{x\^^) + hi'^ajif{xf\y{xf^)) (7.3) 

1=1 

The weighted average of the slope over the interval [xi,Xi-\.i\ is then a 
convex combination of the slopes f{x^P,y{x\^'^)]j = 1,...,M, given by 

M M 

1=1 i=l 

Equation (7.2) finally becomes 

M 

y(x^"P « y(xpp-h/li ^ 6j/(a;pP y(a;pp) 

j=i 


(7.4) 
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The formulas (7.3) and (7.4) define the M-stage Runge-Kutta method. 

If the coefhcients Qji = 0 for all l > j the resulting Runge-Kutta method is 

explicit. If ttji = 0, for all l > j, but ajj + 0 for some j, the method is 

semi-explicit. Otherwise, it is implicit [14]. 

The quantity Ii = y(xf ^)-t-defines 

the local truncation error of the M-stage Runge-Kutta method. The M-stage 

Runge-Kutta method is said to be of order P £ N, if its local truncation error 

is of (!l(h^+^), where/i = max ihA. 

i=0,...,N-l 

The order of the M-stage Runge-Kutta method is achieved by satisfying 
conditions on the coefhcients {a)ij,bi and Cj, i,j £ {1,...,M}. Butcher [14] 
stated the number of conditions on an implicit M-stage Runge-Kutta method, 
to be of order P as in table (7.1). 

TABLE 7.1: The number of restrictions on a P-th order implicit Runge-Kutta 
method 


Order (P) 

1 

2 

3 

4 

5 

6 

7 

8 

Number of Restrictions 

1 

2 

4 

8 

17 

37 

85 

200 


In Table (7.2) we state the conditions on the coefhcients {A,b,c) for the 
M-stage Runge-Kutta methods up to order four. Here a method can of order 
P if it satishes all the conditions from 1 to P. 

TABLE 7.2: The relationships between the order of the M-stage Runge-Kutta 
method, and the conditions on the triplet (A, 6,c) 


Order 

Conditions 

1 


2 

1 

l^i=l 2^7 = 1 — 2 

3 

h..2_l 
— 3 

v-^A<Z 7 1 

2^7=1 OidijCj — g 

4 

Z^z=l — 4 

y-M y^M L c- — - 

v-^Ai v-^A(Z 7 1 

2^i=l 2^j = l — 12 

7 \ 1 


7.2 Explicit Runge-Kutta Methods 

An M-stage Runge-Kutta method is explicit if its Butcher table satishes 
Qij = 0 for all j > i. That is 
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A 

TT- 


0 

0 

0 

0 

0 

C2 

021 

0 

0 

0 

C3 

«31 

«32 

0 

0 

Cm 

omi 

a-M2 

■■■ «MM-l 

0 


h 

b2 

hM-l 

bu 


In an explicit Runge-Kutta method, the function slope at a point depends 
only on the slopes at previous points x^^\...,x^^ The most important 
explicit Runge-Kutta methods are the Euler’s methods, Heun’s method 
and the classical fourth-order Runge-Kutta method [16]. 


7.2.1 Euler’s Method 

The simplest explicit Runge-Kutta method is the Euler method, whose 
Butcher tabular is: 


c 

A 0 

0 


- 

1 


and is derived from: 

y(xi+i) = y(xi) + hif(xi,y(xi)) + 0(h^) 
as 

=y^ + h^f(xi,y(xt)) (7.5) 

where y® « y{xi) and h = m.a,yi{hi,i = 1,... ,N}. 

Because the local truncation error in Euler’s method is 0{h?), the approx- 
imation error of the method is 0{h), as the summation of the local truncation 
errors of the N subintervals is 0{N ■ h?) = 0{h). 

Example 7.1 (solving initial value problem with Euler’s method) In 

this example, the Euler’s method will be used to solve the initial value problem 

=t-2ty, y{0) = 1, for te [0,2]. 

To apply Euler’s methods, the interval [0,2] is divided into N subintervals 
(each of length h = 2/N) by the points to = 0 < ti < ... < tjv = 2, where 
tj = j ‘h and tj+i — tj = h = 2/N. At a point tj, the solution function y(tj) is 
approximated by yj. 

Starting from the initial point (to,yo) = (0;1): Euler’s methods use the 
difference formula: 

Vj+i =yj + hf{tj,yj) =yj+h{tj-2tjyj), j = 0,1,..., A^-1 

to approximate y at tj_|-i giving j/j+i. 

The exact solution of the initial value problem 

y{t) = t-2ty, y(0) = 1 
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is 

and the error in approximating y(tj) by yj is 


Error = max{\y{tj) - yj\,j = 0,1,... ,N}. 


The Python code SolvelVPWithEuler solves the given initial value prob- 
lem using Euler’s method: 


1 

# SolvelVPWithEuler . py 


2 

from numpy import linspace, zeros, exp, inf 


3 

from numpy.linalg import norm 


4 

def EulerIVP{f, a, b, N, yO): 


5 

t = linspace(a, b, N+1) 


6 

yexact = 0 .5*{1+exp(-t**2)) 


7 

f = lambda t, y: t-2*t*y 


8 

h = 2.0/N 


9 

y = zeros((N+1,), 'float' ) 


10 

y[0] = yO 


11 

for j in range (N): 


12 

Y[j+1] = y[j] + h*f(t[j], y[j]) 


13 

return t, y, norm(y-yexact, inf) 


14 



15 

Rows = 15 


16 

f = lambda t, y: 2*t*y 


17 

a, b = 0.0, 2.0 


18 

yO = 1.0 


19 

EulerErrors = zeros((Rows, 2), 'float') 


20 

print ( '-' ) 


21 

print ( ' N\t\t Error') 


22 

print ( '-' ) 


23 

for j in range (Rows): 


24 

N = 2**(4 + j) 


25 

[t, y. Error] = EulerIVP(f, a, b, N, yO) 


26 

EulerErrors[j, 0] = N 


27 

EulerErrors[j, 1] = Error 


28 

print ('{0:8.Of}' . format (N), ' \t' , '{0:1.12e}' . format (Error)) 

29 

print ( '-' ) 


30 



31 

RatesOfConvergence = zeros((Rows-1, 2), 'float') 


32 

print ('Rates of convergence of Eulers method:') 


33 

print ( '-' ) 


34 

print { ' N\t\t Conv. Rate') 


35 

print ( '-' ) 


36 

for j in range (Rows-1): 


37 

RatesOfConvergence[j, 0] = EulerErrors[j, 0] 


38 

RatesOfConvergence[j, 1] = log2(EulerErrors[j, 



1]/EulerErrors[j+1, 1]) 


39 

print ('{0:6.Of}'. format (RatesOfConvergence[j, 0]) 


40 

'{0:1.3f}' . format (RatesOfConvergence[j, 1])) 


41 

print ( '-' ) 
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By executing the code the following results are obtained. 

runfileC’D:/PyFiles/SolvelVPWithEuler.py’, wdir=’D:/PyFiles’) 


N 

Error 

16 

2.212914947992e-02 

32 

1.061531476368e-02 

64 

5.201658072192e-03 

128 

2.570740890148e-03 

256 

1.278021559005e-03 

512 

6.371744244492e-04 

1024 

3.181315727203e-04 

2048 

1.589510709855e-04 

4096 

7.944690307737e-05 

8192 

3.971629161525e-05 

16384 

1.985635490032e-05 

32768 

9.927729750725e-06 

65536 

4.963752954890e-06 

131072 

2.481848491165e-06 

262144 

1.240917262835e-06 


Rates of convergence of Eulers method: 

N 

Conv. Rate 

16 

1 

.060 

32 

1 

.029 

64 

1 

.017 

128 

1 

.008 

256 

1 

.004 

512 

1 

.002 

1024 

1 

.001 

2048 

1 

.001 

4096 

1 

.000 

8192 

1 

.000 

16384 

1 

.000 

32768 

1 

.000 

65536 

1 

.000 

131072 

1 

.000 


From the table of order of convergence, it can be seen that Euler method 
is a first-order method {0{h)). 









Explicit Runge-Kutta Methods 
The MATLAB code is: 


171 


1 

ciear ; clc ; 


2 

f = @(t, y) t-2 *t*y ; 


3 

a=0.0;b=2.0;y0=l; 


4 

yexact = @(t) 0.5* ( 1+exp (-t."2)) ; 


5 

Rows =15 ; 


6 

EulerErrors = zeros (Rows, 2) ; 


7 

fprintf ( '- 

-An') ; 

8 

fprintf(' N\t\t\t Error\n') ; 


9 

fprintf ( '- 

-An') ; 

10 

for j = 1 : Rows 


11 

N = 2~(3+j) ; 


12 

[t, y. Error] = EulerIVP{f, a, b, N, yO) 

; 

13 

EulerErrors(j, 1) = N ; 


14 

EulerErrors(j, 2) = Error ; 


15 

fprintf (' %8i\t%l.10e\n' , N, Error) ; 


16 

end 


17 

fprintf ( '- 

\n') ; 

18 



19 

fprintf ( '- 

\n\n') ; 

20 

RatesOfConvergence = zeros (Rows-1, 2) ; 


21 

fprintf (’ Rates of convergence of Eulers method:\n') ; 

22 

fprintf ( '- 

-An') ; 

23 

fprintf ( ' N\t\t Conv. Rate\n' ) ; 


24 

fprintf ( '- 

\n') ; 

25 

for j = 1 : Rows - 1 


26 

RatesOfConvergence(j, 1) = EulerErrors(j, 

1 ) ; 

27 

RatesOfConvergence(j, 2) = log2 (EulerErrors(j, ... 


2)/EulerErrors(j+1, 2)) ; 


28 

fprintf (' %8i\t %1.12e\n', RatesOfConvergence(j, 1), ... 

29 

RatesOfConvergence(j, 2)) ; 


30 

end 


31 

fprintf ( '- 

\n') ; 

32 



33 

function [t, y. Error] = EulerIVP(f, a, b, N, 

yO) 

34 

t = linspace(a, b, N+1) ; 


35 

h = (b-a)/N ; 


36 

y = zeros (1, N+1) ; 


37 

yexact = 0.5* (1+exp (-t.''2) ) ; 


38 

y(i) = yO ; 


39 

for j = 1 : N 


40 

y ( j+l) = y (j) + h*f (t (j) , y ( j) ) ; 


41 

end 


42 

Error = norm (y-yexact, inf) ; 


43 

end 



7.2.2 Heun’s Method 


Heun’s method has the Butcher tabular: 


c 


A 

F 


0 

0 

0 

1 

1 

0 


i 

i 


2 

2 
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Hence, the Heun’s method is of the form: 

(/ {xi,y'^) + f {xi+i,y'^ + hif{xi,y'^))) 

The following MATLAB code solves Example 7.1 using Heun’s method: 


1 

ciear ; clc ; 


2 

f = @(t, y) t-2*t*y ; 


3 

a = 0.0 ; b = 2.0 ; yO = 1 ; 


4 

yexact = @(t) 0.5* ( 1+exp (-t."2)) ; 


5 

Rows = 15 ; 


6 

HeunErrors = zeros (Rows, 2) ; 


7 

fprintf ( '- 

-An') ; 

8 

fprintf(' N\t\t\t Error\n') ; 


9 

fprintf ( '- 

\n') ; 

10 

for j = 1 : Rows 


11 

N = 2-(3+j) ; 


12 

[t, y. Error] = HeunIVP{f, a, b, N, yO) 

; 

13 

HeunErrors(j, 1) = N ; 


14 

HeunErrors(j, 2) = Error ; 


15 

fprintf (' %8i\t%l.10e\n' , N, Error) ; 


16 

end 


17 

fprintf ( '- 

\n\n') ; 

18 

RatesOfConvergence = zeros (Rows-1, 2) ; 


19 

fprintf (’ Rates of convergence of Heuns method:\n') ; 

20 

fprintf ( '- 

—\n') ; 

21 

fprintf ( ' N\t\t Conv. Rate\n' ) ; 


22 

fprintf ( '- 

—\n') ; 

23 

for j = 1 : Rows - 1 


24 

RatesOfConvergence(j, 1) = HeunErrors(j, 

1 ) ; 

25 

RatesOfConvergence(j, 2) = log2 (HeunErrors(j, ... 


2)/HeunErrors(j+1, 2)) ; 


26 

fprintf ('%8i\t %1.12e\n', RatesOfConvergence(j, 1), ... 

27 

RatesOfConvergence(j, 2)) ; 


28 

end 


29 

fprintf ( '- 

\n') ; 

30 



31 

function [t, y. Error] = HeunIVP(f, a, b, N, 

yO) 

32 

t = linspace(a, b, N+1) ; 


33 

h = (b-a) /N ; 


34 

y = zeros (1, N+1) ; 


35 

yexact = 0,5*( 1+exp (-t."2)) ; 


36 

y(i) = yO ; 


37 

for j = 1 : N 


38 

kl = f (t(j) , y(j) ) ; 


39 

k2 = f (t(j + l), y(j)+h*kl) ; 


40 

y(j + l) = y(j) + h/2* (kl+k2) ; 


41 

end 


42 

Error = norm (y-yexact, inf) ; 


43 

end 
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By executing this code, the following results are obtained. 


N 

Error 

16 

1.6647420306e-03 

32 

3.8083683419e-04 

64 

9.1490315453e-05 

128 

2.2439310337e-05 

256 

5.5575532305e-06 

512 

1.3830109670e-06 

1024 

3.4496196688e-07 

2048 

8.6142039946e-08 

4096 

2.1523228200e-08 

8192 

5.3792741372e-09 

16384 

1.3446276315e-09 

32768 

3.3613245520e-10 

65536 

8.4030005176e-ll 

131072 

2.1010082563e-ll 

262144 

5.2486903712e-12 


Rates of 

convergence of Heuns method: 

N 

Conv. Rate 

16 

2.128053707149e+00 

32 

2.057482078881e+00 

64 

2.027590701520e+00 

128 

2.013506569375e+00 

256 

2.006637264733e+00 

512 

2.003303382587e+00 

1024 

2.001647915261e+00 

2048 

2.000823009405e+00 

4096 

2.000411061126e+00 

8192 

2.000204811383e+00 

16384 

2.000104948066e+00 

32768 

2.000053370390e+00 

65536 

1.999822741673e+00 

131072 

2.001052433088e+00 


The table of rates of convergence shows that Heun’s method is a second- 
order method. 
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The Python code is: 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 


from numpy import linspace, zeros, exp, inf, log2 
from numpy.linalg import norm 
def HeunIVP(f, a, b, N, yO): 
t = linspace(a, b, N+1) 
yexact = 0.5* (1+exp (-t**2)) 
h = 2,0/N 

y = zeros ( (N+1,), ’float' ) 

y[0] = yO 

for j in range (N): 

kl = f {t[j] , y[j] ) 
k2 = f{t[j+l], y[j]+h*kl) 
y[j+l] = y[j] + h/2.0*{kl + k2) 
return t, y, norm(y-yexact, inf) 

Rows = 15 

f = lambda t, y: t-2*t*y 
a, b = 0.0, 2.0 
yO = 1.0 

HeunErrors = zeros((Rows, 2), 'float') 

print ( '-' ) 

print ( ' N\t\t Error') 

print ( '-' ) 

for j in range (Rows): 

N = 2** (4 +j) 

[t, y. Error] = HeunIVP(f, a, b, N, yO) 

HeunErrors[j, 0] = N 
HeunErrors[j, 1] = Error 

print (' {0:8.Of} ’. format (N), '\t', '{0:1.12e} format (Error)) 

print ( '-\n ' ) 

RatesOfConvergence = zeros((Rows-1, 2), 'float') 
print ('Rates of convergence of Heuns method:') 

print ( '-' ) 

print ( ' N\t\t Conv. Rate') 

print ( '-' ) 

for j in range (Rows-1): 

RatesOfConvergence[j, 0] = HeunErrors[j, 0] 

RatesOfConvergence[j, 1] = log2(HeunErrors[j, ... 

1]/HeunErrors[j+1, 1]) 

print ('{0:6.Of}' . format (RatesOfConvergence[j, 0]), '\t\t',\ 

'{0:1.3f}' . format (RatesOfConvergence[j, 1])) 
print ( '-' ) 


7.2.3 The Fourth-Order Runge-Kutta Method 


The classical fourth-order Runge-Kutta method has the Butcher tabular: 


c 


A 


0 

0 

0 

0 

0 

1 

2 

1 

2 

0 

0 

0 

f 

2 

0 

1 

2 

0 

0 

1 

0 

0 

1 

0 


i 

2 

2 

i 


6 

6 

6 

6 
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Example 7.2 (Lotka-Voltera model) In this example the fourth-order 
Runge-Kutta method will be used to solve the following predator-prey model: 

x{t) = 0.7x{t) — 1.3x{t)y{t),x{0) = 0.9 
y(t) = x(t)y{t)-y(t), y{0) = 0.1 


with t G [0,50]. 

The MATLAB code for solving the problem of this example is as follows: 


1 

% SolvePredPrey . m 


2 

ciear ; clc ; 


3 

f = @ (t, z) [0. 1 * 2 . (1) -1 . 

3*z(1)*z(2); 

4 

-Z (2)+z (1) *z (2) ] ; 


5 

a=0.0 ; b=50 ; y0= 

o 

o 

6 

N = 500 ; 


7 

X = linspace (a, b, N+1) 

; 

8 

h = (b-a) /N ; 


9 

y = zeros(2, N+1) ; 


10 

y(:, 1) = yO ; 


11 

for j = 1 : N 


12 

kl = f (x(j) , y(:, j) 

) ; 

13 

k2 = f(x(j)+h/2, y ( ; 

, j)+h/2*kl) ; 

14 

k3 = f(x(j)+h/2, y(; 

, j)+h/2*k2) ; 

15 

k4 = f(x(j)+h, y(:. 

j)+h*k3) ; 

16 

y(:, j + 1) = y(:, j) 

+ h/6.0*{kl+2*k2+2*k3+k4) ; 

17 

end 


18 

figure (1) ; 


19 

plot {x, y(l, :), '-b' , x 

, y(2, :), '-.m', 'LineWidth', 2) 

20 

xlabel ( 'Time (t) ' ) ; 


21 

ylabel ( ’Population' ) ; 


22 

legend ( 'Prey' , 'Predator') ; 

23 

set (gea, 'FontSize', 12) 


24 

set {gea, 'Fontweight' , 'bold') ; 

25 

grid on ; 


26 

set (gea, 'XTick', linspace (a, b, 11)) ; 

27 

set (gea, 'YTick', 0:0.3: 

3) ; 

28 

set (gea, 'GridLineStyle' 

, ' : ') 

29 

grid on ; 



Executing the code SolvePredPrey shows in the graph in Figure 7.1. 
The Python code is: 


1 from numpy import linspace, zeros, arange, array 

2 import matplotlib.pyplot as plt 

3 f = lambda t, z: array ([ 0.7*z [ 0 ] -1. 3* z [ 0 ] *z [1], z[0]*z[l] - z[l]]) 

4 a, b, N = 0,0, 50.0, 500 

5 h = (b-a)/N 

6 X = linspace(a, b, N+1) 

7 y = zeros((l+N, 2), 'float' ) 

8 y[0, 0] , y[0, 1] = 0.9, 0.1 

9 for j in range (N): 

10 kl = f (x[ j] , y[ j, ; ] ) 

11 k2 = f(x[j]+h/2, y[j, :]+h/2*kl) 
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Time (t) 


FIGURE 7.1: Dynamics of the predator and prey populations obtained by the 
fourth-order Runge-Kutta method. 


12 k3 = f(x[j]+h/2, Yijr :]+h/2*k2) 

13 k4 = f(x[j + l], Yijf :]+h*k3) 

14 y[j + l/ :] = y[j, :] + h/6 . * {kH-2 . *k2+2 . *k3+k4) 

15 plt.figure(1, figsize=(14, 14)) 

16 plt.plot(x, y[:, 0], color='b', ls='-', lw= 2, label= 'Prey' ) 

17 plt.plot(x, y[:, 1]^ color= 'm' ,ls= '-.' , lw= 2, label= 'Predator' ) 

18 plt.xticks(arange(a, b+0.5, 5), fontweight= ’bold' ) 

19 plt.yticks(arange(0, 3.3, 0.3), fontweight= 'bold' ) 

20 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

21 plt.ylabel( 'Population' , fontweight= 'bold' ) 

22 plt.grid(True, ls=':') 

23 plt. legend () 


7.3 Implicit Runge-Kutta Methods 

This section discusses some implicit and collocation Runge-Kutta methods for 
solving the initial value problems (7.1). 

An M-stages Runge-Kutta method is implicit, if in its Butcher tablular 
Ojj + 0 for some j > i. 


7.3.1 The Backward Euler Method 


The simplest implicit Runge-Kutta method is the backward Euler method. 
The Butcher table for the backward Euler method is as follows: 


1 

1 


1 
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y-+i=y- + /i/(x„,y-+i) 

The backward Euler method is a first order method with a truncation error 
of 0{h) where h is the maximum step size between two mesh points Xj and 

Xj-\-l. 

It can be seen that the evaluation of the solution function y{x) at a; = Xn+i 
requires the evaluation of the slope function at the same point {x = Xn+i)- To 
avoid this problem, Euler method can be used for giving an estimate of 
for the right-hand side. 

The following Python code solves example 7.1: 


1 from numpy import linspace, zeros, exp, inf, log2 

2 from numpy.linalg import norm 

3 def BackwardEulerIVP(f, a, b, N, yO): 

4 t = linspace(a, b, N+1) 

5 yexact = 0.5* (1+exp (-t**2)) 

6 h = 2.0/N 

7 y = zeros((N+1,), ’float' ) 

8 y[0] = yO 

9 for j in range (N): 

10 Y[j+1] = y[j] + h*f{t[j], y [ j]+h*f (t [ j] , y[j])) 

11 return t, y, norm(y-yexact, inf) 

12 

13 Rows = 15 

14 f = lambda t, y: t-2*t*y 

15 a, b = 0.0, 2.0 

16 yO = 1.0 

17 BackwardEulerErrors = zeros((Rows, 2), 'float') 

18 print ( '-' ) 

19 print ( ' N\t\t Error') 

20 print ( '-' ) 

21 for j in range (Rows): 

22 N = 2**(4 + j) 

23 [t, y. Error] = BackwardEulerIVP(f, a, b, N, yO) 

24 BackwardEulerErrors[j, 0] = N 

25 BackwardEulerErrors[j, 1] = Error 

26 print (' {0:8.Of} format (N), '\t', '{0:1.12e} format (Error)) 

27 print ( '-\n ' ) 

28 

29 RatesOfConvergence = zeros((Rows-1, 2), 'float') 

30 print ('Rates of convergence of BackwardEulers method:') 

31 print ( '-' ) 

32 print ( ' N\t\t Conv. Rate') 

33 print ( '-' ) 

34 for j in range (Rows-1): 

35 RatesOfConvergence[j, 0] = BackwardEulerErrors[j, 0] 

36 RatesOfConvergence[j, 1] = log2(BackwardEulerErrors[j, ... 

1]/BackwardEulerErrors[j + 1, 1]) 

37 print ( '{0:6.Of}' . format (RatesOfConvergence[j, 0]), '\t\t',\ 

38 '{0:1.3f} '. format (RatesOfConvergence[j, 1])) 

39 print ( '-' ) 

40 import matplotlib.pyplot as plt 
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41 plt.figure(1, figsize=(10, 10)) 

42 plt.plot(t, y, color= ’purple' ,lw=2, label='$y(x) = ... 

0.5(l+e"{-x-2})$') 

43 plt.xlabel ( 'X' , fontweight= 'bold' ) 

44 plt.ylabel( 'y' , fontweight= 'bold’ ) 

45 plt.grid(True, ls=':') 

46 plt.xticks(arange(0.0, 2.2, 0.2), fontweight= 'bold' ) 

47 plt.yticks(arange(0.5, 1.05, 0.05), fontweight= 'bold' ) 

48 plt. legend () 


By executing this code, the following results are obtained. 

runfileC’D;/PyFiles/SolvelVPWithBackwardEuler.py’, 
wdir=’D:/PyFiles’) 


N 

Error 

16 

4.337814100757e-02 

32 

2.037832192200e-02 

64 

9.905832986266e-03 

128 

4.886733400528e-03 

256 

2.427510216728e-03 

512 

1.209810748381e-03 

1024 

6.039264582937e-04 

2048 

3.017197337041e-04 

4096 

1.507990196000e-04 

8192 

7.538431169796e-05 

16384 

3.768835804929e-05 

32768 

1.884322987655e-05 

65536 

9.421377654029e-06 

131072 

4.710629515681e-06 

262144 

2.355299925094e-06 


Rates of convergence of BackwardEulers method 

N 

Conv. Rate 

16 

1, 

.090 

32 

1, 

.041 

64 

1, 

.019 

128 

1, 

.009 

256 

1, 

.005 

512 

1, 

.002 

1024 

1, 

.001 

2048 

1, 

.001 

4096 

1, 

.000 

8192 

1, 

.000 
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FIGURE 7.2: Solutiori of the initial value problem of Example 7.1, given by 
y{x) = (l + e“'^^)/2. 


16384 

1, 

.000 

32768 

1, 

.000 

65536 

1. 

.000 

131072 

1, 

.000 


The graph of the problem solution is shown in Figure 7.2. 
The MATLAB code is: 


1 ciear ; clc ; 

2 f = @(t, y) t-2*t*y ; 

3 a=0.0;b=2.0;y0=l; 

4 yexact = @(t) 0.5* {l+exp (-t.''2) ) ; 

5 Rows = 15 ; 

6 BackwardEulerErrors = zeros(Rows, 2) ; 

7 fprintf ( '-\n ' ) ; 

8 fprintf ( ' N\t\t\t Error\n') ; 

9 fprintf ( '- \n ' ) ; 

10 for j = 1 : Rows 

11 N = 2 ^ (3+ j ) ; 

12 [t, y. Error] = BackwardEulerIVP(f, a, b, N, yO) ; 

13 BackwardEulerErrors(j, 1) = N ; 

14 BackwardEulerErrors(j, 2) = Error ; 

15 fprintf (' %8i\t%l.10e\n' , N, Error) ; 

16 end 

17 fprintf ( '-\n\n') ; 

18 


19 

plot {t, y, 

. 'm', 'LineWidth', 2) 

20 

xlabel('x') ; ylabel(’y') ; 

21 

grid on ; 


22 

set (gea, ' 

'XTick', linspace(a, b 

23 

set (gea, ' 

'YTick', 0.5:0,05:1.0) 

24 

set (gea, ' 

' f ontweight' , 'bold') 


25 legend('y(x) = (l+e''{-x''2})/2' ) ; 

26 

27 function [t, y. Error] = BackwardEulerIVP(f, a, b, N, yO) 
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28 t = linspace(a, b, N+1) ; 

29 h = (b-a) /N ; 

30 y = zeros(l, N+1) ; 

31 yexact = 0.5* (1+exp (-t."2)) ; 

32 y (1) = yO ; 

33 for j = 1 : N 

34 y(j+i) = y(j) + h*f{t{j), y (j)+h*f (t (j), y(j))) ; 

35 end 

36 Error = norm (y-yexact, inf) ; 

37 end 


7.3.2 Collocation Runge-Kutta Methods 

Collocation methods are implicit Runge-Kutta methods, but not ali implicit 
Runge-Kutta methods are collocation methods. 

To advance the solution of the differential equation from a point Xn to a 
point Xn+i, a collocation method constructs a polynomial Pm{x) of degree M 
that interpolates the solution of the ODE at collocation points Xn + Cjh, j = 
l,... ,M, with 0 < Cj < 1, j = 1,..., M. That is 

P'M{Xn + Cjh) = f{Xn + Cjh,PMiXn + Cjh)), j = 

and satisfies, 

PM{xi) =y\ i = 0,l,...,n. 

Collocation Runge-Kutta methods are deeply related to the Gauss quadra¬ 
ture methods discussed in the past chapter. The collocation are obtained by 
transforming Gauss points (in [—1,1]) such as the LG, LGL, LGR, ...etc points 
to the interval [0,1], using the transformation 

7.3.2.1 Legendre-Gauss Methods 

The first four Gauss-Legendre methods have the Butcher’s tables: 

(i) M = l: 


1 

1 

2 

2 


1 


This method is called the mid-point rule. 
(ii) M = 2: 


1 V3 

1 1 V3 

2 6 

4 4 6 

1 , +3 

1 , +3 1 

2 "T fi 

4 6 4 


1 1 


2 2 
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(iii) M = 3 : 


1 

VTE 

5 

2 

vTs 

5 

vTs 

2 

10 

36 

9 

10 

36 

30 


1 

5 , vT5 


2 

5 

vTs 


2 

36 24 


9 

36 

24 

1 

, vTs 

5 , vT5 

2 

, vTs 


5 

2 

+ 10 

36 30 

9 

+ 15 


36 


15 

18 


4 

9 


5 

18 


Example 7.3 In this example the mid-point rule will be used to solve the 
logistic-growth model: 

P{t)=rP{t){l-P{t)),P{0) = Po,te[0,T] 

The exact solution of the model is 

p(t) =_ — _ 

Po + (l-Po)e-'-‘ 

The Python code for solving the logistic growth model with r = 0.2,Po = 
0.1 in [0,30] is: 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 


from numpy import linspace, zeros, exp, inf, log2, arange 
from numpy.linalg import norm 
def GaussMidIVP(f, a, b, N, PO): 
t = linspace(a, b, N+1) 

Pexact = PO/{P0+(1-PO)*exp {-0.2*t)) 
h = (b-a)/N 

P = zeros { {N+1,), 'float') 

P[0] = PO 

for j in range (N): 

Ph = P[j] + h/2*f(t[j], P[j]) 

P[j+1] = P[j] + h*f {t[j]+h/2, Ph) 
return t, P, norm(P-Pexact, inf) 


Rows = 12 
r = 0.2 

f = lambda t, P: r*P*(l-P) 
a, b = 0.0, 30.0 
PO = 0.1 

GaussMidErrors = zeros((Rows, 2), 'float') 

print ( '-' ) 

print ( ' N\t\t Error') 

print ( '-' ) 

N = 100 

for j in range (Rows): 

[t, P, Error] = GaussMidIVP(f, a, b, N, PO) 

GaussMidErrors[j, 0] = N 
GaussMidErrors[j, 1] = Error 

print ('{0:8.Of}' . format (N), ' \t ' , '{0:1.12e}' . format (Error) ) 

N *= 2 

print ( '-\n ' ) 


31 
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32 RatesOfConvergence = zeros((Rows-1, 2 ), 'float') 

33 print( 'Rates of convergence of Gauss Midpoint method:') 


34 print ( ’-' ) 

35 print ( ' N\t\t Conv. Rate') 

36 print ( '-' ) 


37 for j in range (Rows-1): 

38 RatesOfConvergence[j/ 0] = GaussMidErrors[j, 0] 

39 RatesOfConvergence[j/ 1] = log2(GaussMidErrors[j, ... 

1]/GaussMidErrors[j + 1, 1]) 

40 print ( '{0:6.Of}' . format {RatesOfConvergence[j, 0]), '\t\t',\ 

41 '{0:1. 3f}'. format (RatesOfConvergence[j, 1])) 

42 print ( '-' ) 

43 

44 import matplotlib.pYplot as plt 

45 plt.figure(1, figsize=(10, 10)) 

46 plt.plot(t, P, color=’purple' ,lw=2, label='$P(t) = ... 

P_0/ (P_0+ (1-P_0 ) e''{-rt}) $ ’ ) 

47 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

48 plt.ylabel( 'Population (P(t))', fontweight= 'bold' ) 

49 plt.grid(True, ls=':') 

50 plt.xticks (arange(a, b+(b-a)/10, (b-a)/10), fontweight= 'bold' ) 

51 plt.yticks(arange(0.0, 1.05, 0.1), fontweight= ’bold' ) 

52 plt. legend () 

By executing the code, the errors and rates of convergence for different 
values of mesh points are shown below. 

runfile(^D;/PyFiles/SolveLogisticWithMidPoint.py^, 
wdir= ;/PyFiles ^) 


N 

Error 

100 

1.948817140496e-05 

200 

4.811971295204e-06 

400 

1.195559782174e-06 

800 

2.979672797387e-07 

1600 

7.437647475683e-08 

3200 

1.857971743124e-08 

6400 

4.643131656934e-09 

12800 

1.160556650781e-09 

25600 

2.901111573195e-10 

51200 

7.252876077501e-ll 

102400 

1.812816563529e-ll 

204800 

4.535261055594e-12 

409600 

1.134092819655e-12 

819200 

2.786659791809e-13 

1638400 

6.639133687258e-14 
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N 

Conv. Rate 

100 

2.018 

200 

400 

2.009 

2.004 

800 

2.002 

1600 

2.001 

3200 

2.001 

6400 

12800 

2.000 

2.000 

25600 

2.000 

51200 

2.000 

102400 

1.999 

204800 

2.000 

409600 

819200 

2.025 

2.069 


The MATLAB code is: 


1 ciear ; clc ; 

2 Rows =15 ; 

3 r = 0.2 5 ; 

4 f = @(t, P) r*P*(1-P) ; 

5 a=0.0;b=20.0; 

6 PO = 0.2 ; 

7 GaussMidErrors = zeros(Rows, 2) 


8 fprintf ( '- 

9 fprintf ( ' 

10 fprintf ( ' - 

11 N = 100 ; 

12 for j = 1 

\n' ) ; 

N\t\t Error\n') ; 

\n') ; 

: Rows 


13 [t, P, Error] = GaussMidIVP(f, a, b, N, PO) 

14 GaussMidErrors(j, 1) = N ; 

15 GaussMidErrors(j, 2) = Error ; 

16 fprintf ( '%8i\t%l.12e\n' , N, Error) ; 


17 N = 2 

18 end 

19 fprintf ( ' - 

20 

21 f igure (1) 

* N ; 

- \n') 


22 plot(t, P, '-m', 'LineWidth', 2) ; 

23 legend('P(t) = P _0 / (P_0 + (1-P _0 ) e" {-rt}) 

24 xlabel ( 'Time (t)' ) ; 

25 ylabel ( ’Population (P(t))') ; 

26 grid on 


27 set (gea, 

28 set (gea. 

fontweight' , 'bold') ; 
fontsize' , 12) ; 


29 set (gea, 'XTick', a: (b-a)/10:b) 
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30 set (gea, 'YTick', 0:0.1:1) ; 

31 

32 RatesOfConvergence = zeros (Rows-1, 2) ; 

33 fprintf ('Rates of convergence of Gauss Midpoint method:\n') 

34 fprintf ( ’-\n ' ) 

35 fprintf ( ' N\t\t Conv. Rate\n') 

36 fprintf ( '-\n ' ) 

37 for j = 1 : Rows-1 

38 RatesOfConvergence(j, 1) = GaussMidErrors(j, 1) ; 

39 RatesOfConvergence(j/ 2) = log2 (GaussMidErrors(j, ... 

2)/GaussMidErrors(j+1, 2)) ; 

40 fprintf ('%6i\t\t%l.3f\n' , RatesOfConvergence(j, 1), ... 

RatesOfConvergence(j, 2)) ; 

41 end 

42 fprintf ( '-\n ' ) ; 

43 

44 function [t, P, Error] = GaussMidIVP(f, a, b, N, PO) 

45 t = linspace(a, b, N+1) ; 

46 h = (b-a) /N ; 

47 P = zeros{l, N+1) ; 

48 Pexact = PO./(P0+(1-PO) *exp {-0.25+t)) ; 

49 P (1) = PO ; 

50 for j = 1 : N 

51 Ph = P (j) + h/2*f {t {j) , P ( j) ) ; 

52 P(j + 1) = P(j) + h*f {t{j)+h/2, Ph) ; 

53 end 

54 Error = norm (P-Pexact, inf) ; 

55 end 


The solution graph is explained in Figure 7.3. 

7.3.2.2 Lobatto Methods 

There are three kinds of Lobatto methods: Lobatto IIIA, Lobatto IIIB and 
Lobatto IIIC methods. In this section only two Lobatto methods will be used 
for solving the IVP (7.1), both are Lobatto IIIA methods. 



Time (t) 


FIGURE 7.3: Solution of the logistic growth model using the Gauss mid-point 
method. 
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The first method is the implicit trapezoidal method which has the 
Butcher table: 


0 

0 

0 

1 

1 

1 

2 

2 


i 

i 


2 

2 


and has the form: 


+ l {f{tn,yn + f{tn+i , 


The implicit trapezoidal method is of second order. 


Example 7.4 In this example, MATLAB and Python programmes based on 
the implicit trapezoidal rule will be used to solve a System of two ordinary 
differential equations, representing two competing species x{t) and y{t), t G 
[0,r], both are subject to logistic growth model, in the absence of the other 
population: 


dx{t) 

dt 

dy{t) 

dt 


rix{t){l — x{t)) — ax{t)y{t), a;(0) = xq 
r 2 y{t){l - y{t)) - bx{t)y{t), y{0) = yo 


The MATLAB code is 


1 % ITSolCompSpec . m 

2 ciear ; clc ; 

3 t0=0 ; T=50.0 ; rl=0.3 ; r2=0,2 ; a=1.0 ; b=0.9 ; 

4 f = @(t, z) [0.3*z (1)* (1-z (1))-z (1)*z (2); ... 

0.2*z (2)* (1-z(2))-0.9*z(l)*z{2)] ; 

5 zO = [0.5; 0.5] ; Dim = length{z0) ; 

6 N = 100*ceil (T-tO) ; 

7 t = linspace{tO, T, N+1) ; 

8 h = (T-tO)/N ; 

9 Epsilon = le-15 ; 

10 n = length (t) ; 

11 z = zeros (Dim, N+1) ; 

12 YPrime = zeros (n, Dim) ; 

13 z { : , 1) = z 0 ; 

14 i = 1 ; 

15 while (i < N) 

16 kl = f (t {i) , z { : , i) ) ; 

17 k2 = feval{f, t(i + l),z{:, i)+h*kl) ; 

18 YPrime (i, :) = kl ' ; 

19 yest = z{:, i) + h*kl ; 

20 z(:, i + 1) = z{:, i) + h/2.0* (kl+k2) ; 

21 while(norm (z(:, i+1)-yest)>Epsilon) 

22 yest = z(:, i + 1) ; 

23 k2 = feval{f, t (i+1),yest) ; 

24 z(:, i + 1) = z(:, i) + {h/2.0) * (kl+k2) ; 


25 


end 
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Time (years) 

FIGURE 7.4: Solution of the competition model based on implicit trapezoidal 
rule solver. 


26 i = i + 1 ; 

27 end 

28 X = z (1, : ) ; 

29 y = z ( 2, : ) ; 

30 plot(t, X, '-b' , t, Y, ':m', 'LineWidth', 2) ; 

31 grid on 

32 set (gea, 'fontweight' , 'bold') 

33 legend ( ’Population 1', 'Population 2') ; 

34 axis{[tO, T, -0.1, 1.1]) 

35 xlabel('Time (years)’) ; ylabel (’ Population Density') ; 

36 set (gea, 'XTick', 10: (T-tO)/10:T) ; 

37 set (gea, 'YTick', 0:0.1:1) ; 


By exeeuting the code, the solution of the competition model is shown in 
Figure 7.4. 

The Python code is: 


1 import numpy as np 

2 to = 0 ; T = 50.0 ; rl = 0.3 ; r2 = 0.2 ; a = 1.0 ; b = 0.9 ; 

3 f = lambda t, z: np.array([0.3*z[0]*(1-z[0])-z[0]*z[1], ... 

0.2*z[l]*(l-z[l])-0.9*z[0]*z[l]]) 

4 zO = [0.5, 0.5] 

5 Dim = len (z0) 

6 N = 100*int (T-tO) 

7 t = np.linspaee(tO, T, N+1) 

8 h = (T-tO)/N 

9 Epsilon = le-15 

10 n = len (t) 

11 z = np.zeros((Dim, N+l), 'float') 

12 YPrime = np.zeros((n, Dim), 'float') 

13 z [:, 0 ] = z 0 

14 i = 0 

15 while i < N: 

16 kl = f (t [i] , z [:, i] ) 

17 k2 = f (t [i+1] , z [ :, i]+h*kl) 

YPrime[i, :] = kl 


18 
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19 yest = z[:, i] + h*kl 

20 z[:, i+1] = z[:, i] + h/2.0*(kl+k2) 

21 while np.linalg.norm(z[:, i+1]-yest)>Epsilon: 

22 yest = z[:, i + 1] ; 

23 k2 = f {t [i + 1] , yest) ; 

24 z[:, i+1] = z[:, i] + {h/2.0)*(kl+k2) 

25 i = i + 1 

26 

27 X, y = z [0, : ] , z [ 1, : ] 

28 y = z [ 1, : ] 

29 import matplotlib.pyplot as plt 

30 plt.plot(t, X, '-b', lw=2, label= 'Population 1') 

31 plt.plot(t, y, ':m', lw=2, label= 'Population 2’) 

32 plt.grid(True, ls=':') 

33 #set(gea, 'f ontweight ’, 'bold') 

34 plt. legend () 

35 plt.axis([tO, T, -0.1, 1.1]) 

36 plt.xlabel (' Time (years) ', fontweight= 'bold' ) 

37 plt.ylabel (' Population Density', fontweight='bold' ) 

38 plt.xticks(np.arange(tO, T+{T-t0)/10, (T-t0)/10), ... 

fontweight= 'bold' ) 

39 plt.yticks(np.arange(-0.1, 1.1, 0.1), fontweight= 'bold' ) 


The second method is Hermite-Simpson’s method. The Butcher’s table of 
the method is: 

0 0 0 


0 


and has the form: 


_ 5 _ 1 

24 3 


' 24 


12 1 

-f- 

6 3 6 


where = tn + h/2 and « y{tn + h/2). The Hermite-Simpson 

method is a fourth order. 


Example 7.5 The Hermite-Simpson method will be used to solve an HIV 
model described by the following equations: 

= s - ^lTT{t) + rT{t){ 1 - {T{t) + I{t))lT„,ax)-kiT{t)V{t) , T(0) = 1000 

^ = k2T{t)V{t) - /(0) = 0 

= N,ybI{t)-hT{t)V{t)-yvV{t),V{0) = lO-^ 

at 

where s = 10,muT = 0.02,r = 0.03,Tmax = 1500,= 2.4 x 10~^,k2 = 2.0 x 
10-^,pi = 0.26,Ny = 850,pb = 0.24,pv = 2.4 and t £ [0,200]. 
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The MATLAB code is: 


1 ciear ; 

2 t0=0;tf=200; 

3 3=10; 

4 mu_T = 0.02 ; 

5 r = 0.03 ; 

6 Tmax = 1500 ; 

7 kl = 2.4e-5 ; 

8 k2 = 2 . Oe-5 ; 

9 mu-I = 0.26 ; 

10 Nv = 850 ; 

11 NC = 2 00 ; 

12 Q = 2 ; 

13 mu_b = 0.24 ; 

14 mu-V = 2.4 ; 

15 

16 f = @(t, x) [ s-mu-T *x (1)+r*x {1) * (1-(x (1) 4-x (2) )/Tmax)-kl*x {1) *x (3) ; 

17 k2*x {1) *x (3)-mu_I *x (2 ) ; 

18 Nv*mu-b*x (2) -kl*x (1) *x (3) -mu-V*x ( 3) ] ; 

19 xO = [le+3; 0; le-3] ; 

20 N = 10* (tf-tO) ; 

21 Epsilon = le-13 ; 

22 x = zeros(l+N, 3) ; 

23 t = linspace{t0, tf, 1+N) ; 

24 h = {tf-tO) /N ; 

25 X (1, :) = xO ; 

26 

27 for i = 1 : N 

28 Kl = f (t (i) , X {i, : ) ) ; 

29 xest = x(i, :) + h*Kl ' ; 

30 K3 = f(t{i + l), xest) ; 

31 xmid = 0.5*{x{i, :)+xest)+h/8*(Kl'-K3') ; 

32 K2 = f(t{i)+h/2, xmid) ; 

33 x(i+l,:) = x{i,:) + h/6*{Kl'+4*K2'+K3') ; 

34 end 

35 T = x(:, 1) ; I = x(:, 2) ; V = x(:, 3) ; 

36 f igure (1) ; 

37 subplot (3, 1, 1) ; 

38 plot{t, T, '-b' , ' LineWidth ' 2) ; 

39 xlabel ( 'Time' ) ; ylabel ( 'Susceptible CD4+ T-Cells') ; 

40 grid on ; 

41 set (gea, 'fontweight’ , 'bold') ; 

42 set (gea, 'XTick', linspace(0, tf, 11)) ; 

43 grid on ; 

44 subplot (3, 1, 2) ; 

45 plot{t, I, '— r', 'LineWidth', 2) ; 

46 xlabel (' Time' ) ; ylabel ('Infected CD4+ T-Cells’) ; 

47 grid on ; 

48 set (gea, 'fontweight' , 'bold') ; 

49 set (gea, 'XTick', linspace(0, tf, 11)) ; 

50 grid on ; 

51 subplot (3, 1, 3) ; 

52 plot{t, V, '-.m', 'LineWidth', 2) ; 

53 xlabel (' Time' ) ; ylabel ('Viral load') ; 

grid on ; 


54 
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55 set (gea, 'fontweight’ , 'bold') ; 

56 set (gea, 'XTick', linspace{0, tf, 11)) ; 

57 grid on ; 

Exeeuting the above code shows in the graph in Figure 7.5. 
The Python code is: 


1 import numpy as np 

2 tO = 0 

3 tf = 200 

4 S = 10 

5 mu_T = 0.02 

6 r = 0.03 

7 Tmax = 1500 

8 cl = 2.4e-5 

9 c2 = 2 . Oe-5 

10 mu_I = 0.26 

11 Nv = 850 

12 mu_b = 0.24 

13 mu-V = 2.4 

14 

15 f = lambda t, x: ... 

np.array([s-{mu_T+r*(l-(x[0]+x[l])/Tmax)-cl*x[2])*x[0],\ 

16 c2*x [ 0 ] *x [ 2 ] -mu_I *x [ 1 ] , \ 

17 Nv*mu-b*x [l]-cl*x[0]*x[2] -mu-V*x [ 2 ] ] ) 

18 zO = [le+3, 0.0, le-3] ; 

19 Dim = len ( z 0 ) 

20 N = 100*int (tf-tO) 

21 t = np. linspace (tO, tf, N+1) 

22 h = (tf-tO) /N 

23 Epsilon = le-15 

24 n = len (t) 

25 z = np.zeros((Dim, N+1), 'float’) 

26 zmid = np.zeros((Dim, N), 'float') 

27 YPrime = np.zeros((n, Dim), 'float') 

28 z [:, 0 ] = z 0 

29 i = 0 

30 while i < N: 

31 kl = f (t [i] , z [:, i] ) 

32 zest = z[:, i]+h*kl 

33 k3 = f(t[i+l], zest) 

34 zmid[:, i] = 0.5*(z[:, i]+zest) + h/8*(kl-k3) 

35 k2 = f (t [ i ]+h/2.0, zmid [ : , i]) 

36 z[:, i+1] = z[:, i] + h/6.0*(kl+4*k2+k3) 

37 i = i + 1 

38 

39 T, I, V=z[0, :], z[l, :], z[2, :] 

40 import matplotlib.pyplot as plt 

41 plt. figure (1) 

42 plt.plot(t, T, color= 'orangered' , lw=2) 

43 plt.subplot(3, 1, 1) 

44 plt.plot(t, T, color= 'orangered' , lw=2) 

45 plt.xlabel( 'Time' ) 

46 plt.ylabel( 'Susceptible CD4+ T-Cells', fontweight= 'bold' ) 
plt.grid(True, ls=':') 


47 
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FIGURE 7.5: Solutiori of the HIV model obtained by the Hermite-Simpson’s 
method. 


48 plt.xticks(np.linspace(0/ tf, 11)^ fontweight= 'bold' ) 

49 plt.subplot(3, 1 , 2) ; 

50 plt.plot(t, I, color= ’purple' , lw=2) 

51 plt.xlabel( 'Time' ) 

52 plt.ylabel( 'Infected CD4+ T-Cells', f ontweight= 'bold' ) 

53 plt.grid(True, ls=':’) 

54 plt.xticks (np.linspace(0/ tf, 11)) 

55 plt.subplot(3, 1, 3) 

56 plt.plot(t, V, color= ’darkred’ , lw=2) 

57 plt.xlabel( 'Time' , fontweight= 'bold' ) 

58 plt.ylabel (' Viral load', fontweight= 'bold' ) 

59 plt.grid(True, ls=':') 

plt.xticks (np.linspace(0, tf, 11), fontweight= 'bold' ) 


60 
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7.4 MATLAB ODE Solvers 

MATLAB provides many routines to solve Systems of first order differential 
equations. In this lecture we will discuss how the MATLAB function ode45 
can be used for solving a System of first order initial value problems (IVPs) 
of ordinary differential equations. 

7.4.1 MATLAB ODE Solvers 

There are two general classes of ODE solvers in MATLAB: solvers for non-stiff 
problems, and solvers for stiff problems. 

(1) Non-stiff problems solvers: 

MATLAB has three solvers for non-stiff problems: ode23, ode45 and 
odellS. 

The ode23 is of low order that integrates a System of first-order differential 
equations using second and third order explicit Runge-Kutta methods. 

The ode45 is of medium order that integrates a System of first-order odes 
using fourth and fifth order explicit Runge-Kutta methods. It is more 
accurate than ode23 in solving non-stiff problems. 

The ode 113 solver belongs to the Adams-Bashforth-Moulton predictor- 
corrector pairs of orders 1 to 13. Therefore, the odell3 is a low to medium 
ODE solver. In many problems it is more efficient than the ode45 solver. 

(2) Stiff problems solvers MATLAB possesses five solvers for solving stiff 
problems. They are odel5s, ode23s, ode23t, ode23b and odel5i. 

7.4.2 Solving a Single IVP 

We consider an IVP of the form: 



(7.6) 

O 

II 

; ® 

(7.7) 


To solve the IVP (7.6)-(7.7), we have to write a function, which returns the 
right-hand side of Equation (7.6). This can be done as follows: 


1 function z = fp(t, y) 

2 z = f (t, y) ; 

Now from the command window (or a MATLAB script), we can define the 
time period, the initial condition and call the ode45 function to solve the given 
IVP. We do this as follows: 
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1 tspan = [t_0, t_l] ; 

2 IC = yO ; 

3 [t, y] = ode45(@fp, tspan, yO) ; 

4 plot {t, y) ; 


The Symbol @ in front of the name of the function, informs MATLAB that 
what is follows is the name of the function which defines the slope. 

Example 7.6 Use MATLAB to solve the initial value problem: 

^ = 0.0l2/(t)(l-y(t)), tG [0,700], y(0) = 0.2 

To solve the above given IVP, we first define a function LogisticGrowthSlope 
as follows: 


1 % LogisticGrowthSlope . m 

2 function z = LogisticGrowthSlope(t, y) 

3 Z = 0.01*y*(1-y) ; 


Then, a MATLAB script SolveLogisticWithAlee.m to solve the problem 
and show the results is as follows: 


1 

% SolveLogisticWithAlee.m 

2 

ciear ; 

clc ; 

3 

tspan = 

[0, 700] ; 

4 

yO = 0.3 

; 

5 

xO = 0.2 

; 

6 

LogGrowth = @(t, y) 0.01*y*(1-y)*(4*y-l) ; 

7 

[t, y] = 

ode45(LogGrowth, tspan, yO) ; 

8 

[S, X] = 

ode45(LogGrowth, tspan, xO) ; 

9 

plot {t, 

y, '-b' , s, X, '-.r', 'LineWidth', 2) ; 

10 

legend(’y_0 = 0.3 > Threshold = 0.25', 'x_0 = 0.2 < Threshold ... 


= 0. 

25’) ; 

11 

hold on 

; 

12 

plot{[0. 

0, 700], [0.25, 0.25], '—k' ) ; 

13 

xlabel( ' 

Time' ) ; 

14 

ylabel( ’Population' ) ; 

15 

grid on 

; 

16 

axis([0, 

700, 0, 1.1]) ; 

17 

set(gea, 

'fontweight' , 'bold') ; 

18 

set(gea. 

'GridLineStyle' , ; 

19 

set(gea. 

'YTick', 0:0.125:1) ; 


executing the script SolveLogisticWithAlee will show the solution as in 
Figure 7.6: 
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FIGURE 7.6: Solution of the logistic growth model with Alee effect using the 
ode45 MATLAB sol ver. 


7.4.3 Solving a System of IVPs 

In this section, we consider a system of initial value problems. The System is 
of the form: 

^ yi{to) = yi, te[to,ti] 

^ = f2{t,yi{t),...,yn{t)), y2{to) = y2, 

; = ; (7.8) 

^ = fn{t,yi{t),...,yn{t)), yn{to) = yn, teitoAi] 


The first step to solve the above system of IVPs is to write it in a vector form. 
To do that, we let: 


fyiit)\ 


2/2 (t) 


z = 


\yn{t)J 


then, 


dz{t) 


f 2 it,z{t)) 

and z{t(j) = 

1 2/1 (io) \ 
2/2 (io) 


(y\\ 

y% 



\2/n(io)/ 


U/ 


dt 


















194 


Solving Systems of Nonlinear Ordinary Differential Equations 


The second step is to write a function, which can evaluate the right-hand side 
of the System, where the output is an n-dimensional vector. This is done as 
follows: 


1 

function z = 

fp(t, z) 

2 

z = [ f_l (t, 

z) ; 

3 

f_2{t, z) ; 


4 



5 

f _n {t, z) ; 


6 

] ; 



Finally, we write a script to define the initial condition, time space and calls 
the ode45 function to solve the System. 

T = [t_0, t_l] ; 

zO = [y_10; y_20; ...; y_nO] ; 

[t, z] = ode45(@fp, T, zO) ; 

The ode45 function discretizes the time space into m (not equidistant) 

points and returns the resuit in t. Also, the matrix z is of type (m + 1) x (n + 1). 
The first column of z (z(:,l)) is the solution vector yi, the second column 
(z(:,2)) is the solution vector y 2 , ...etc. 

Example 7.7 Using MATLAB, solve: 

S{t) = -0.035/ + 0.02A, 5(0) = 0.85, te [0,300] 
i{t) = 0.0357-0.01/, /(0) = 0.15, te [0,300] 

R{t) = 0.017-0.027?, i?(0) = 0, te [0,300] 

We define a function SIRSSlope.m to evaluate the right-hand sides of the 
System equations. 


1 

% SIRSSlope 


2 

function z = SIRSSlope(t. 

X) 

3 

s = X (1) ; I = X (2) ; 

R = x{3) ; 

4 

z = [-0.03*S*I+0.02*R, 

0.03*S*I-0.01*I; 0.01*1-0.02*R] ; 

5 

end 



We write a MATLAB script 'SolveSIRS.m’ to implement the solver: 


1 % SolveSIRS.m 

2 ciear ; clc ; 

3 tspan = [0, 300] ; 

4 zO = [0.85; 0.15; 0] ; 

5 [t, z] = ode45{OSIRSSlope, tspan, zO) ; 

6 S = z(:, 1) ; I = z(:, 2) ; R=z(:, 3) ; 

7 plot{t, S, '— b', t, I, ':r', t, R, '-.m', 

8 grid on ; 

9 xlabel ( ' Time ' ) ; 


LineWidth', 2) ; 







MATLAB ODE Solvers 


195 


1 

0.9 

0.8 

0.7 

c 0.6 
O 

lo.5 

O. 

o 

Q- 0.4 
0.3 
0.2 
0.1 
0 

0 30 60 90 120 150 180 210 240 270 300 

Time 

FIGURE 7.7: Solution of the SIRS model with the ode45 MATLAB solver. 


“ “ Susceptibles 


. Infected 

—- Recovered 



10 ylabel( ’Population' ) ; 

11 legend (’Susceptibles’ , 'Infected', 'Recovered') ; 

12 axis{[0, 300, 0, 1]) ; 

13 set (gea, 'fontweight' , 'bold') ; 

14 set(gea, 'XTick', linspace{0, 300, 11)) ; 

15 set(gea, 'GridLineStyle' , ':') ; 

The output of exeeuting the script SolveSIRS.m is Figure 7.7. 

7.4.4 Solving Stiff Systems of IVPs 

If the IVP System is stiff, the MATLAB function ode45 fails in finding a 
solution for the problem. An example for such a System is the Vanderpool 
oscillator, given by the second order differential equation: 

y"{x) + a{y^{x)-l)y'{x) + y{x) =0, ?/(0) = 1, y'(0) = 2 

If we let yi{x) = y{x), 2 / 2 ( 2 :) = y'{x), we can write the Vanderpool system can 
be written in the form: 

y'i{x) = 2/2(a;),2/i(0) = 1 

2 / 2 ( 2 ;) = cr(l-2/i(a;))2/2(a;)-2/i(2:),2/2(0) = 2 (7.9) 

The MATLAB ode45 function will fail to solve the Vanderpool system (7.9). 
We can use the MATLAB function, ode23s instead to solve the system (7.9). 
We will declare cr as a global variable and then solve the system as follows: 
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1 %vdpsig.m 

2 function z = vdpsig(t, y) 

3 global sigma 

4 z = zeros {2, 1) ; 

5 z {1) = y (2 ) ; 

6 z{2) = sigma* (1-y (1) '‘2) *y (2)-y {!) ; 


1 % SolveVDPSystem . m 

2 ciear ; clc ; 

3 global sigma ; 

4 tspan = [0200] ; y0= [2; 1] ; 

5 figure(1) ; 

6 sigma = 1 ; 

7 [t,y] = ode23s(Ovdpsig, tspan, yO) ; 

8 subplot (3, 2, 1) ; 

9 plot{t,y(:,1), '-b' , 'LineWidth', 2) ; 

10 xlabel(’x') ; ylabel(’y(x) ') ; 

11 title([ ' \sigma = ' num2str(sigma)]) ; 

12 sigma = 5 ; 

13 [t,y] = ode23s (Ovdpsig, tspan, yO) ; 

14 subplot (3, 2, 2) ; 

15 plot(t,y(:,1), '— b’, 'LineWidth', 2) ; 

16 xlabel('x') ; ylabel(’y(x) ') ; 

17 title ([' \sigma = ' num2str(sigma)]) ; 

18 sigma =10 ; 

19 [t,y] = ode23s(Ovdpsig, tspan, yO); 

20 subplot(3, 2, 3) ; 

21 plot(t,y(:,l), ' —b' , ' LineWidth' , 2) ; 

22 xlabel('x') ; ylabel(’y(x) ') ; 

23 title ([' \sigma = ' num2str(sigma)]) ; 

24 sigma = 20 ; 

25 [t,y] = ode23s(Qvdpsig, tspan, yO); 

26 subplot(3, 2, 4) ; 

27 plot (t, y (:, 1) , ' —b' , ' LineWidth ’ , 2) ; 

28 xlabel('x') ; ylabel(’y(x) ') ; 

29 title ([' \sigma = ' num2str(sigma)]) ; 

30 sigma = 40 ; 

31 [t,y] = ode23s(Ovdpsig, tspan, yO); 

32 subplot (3, 2, 5) ; 

33 plot(t,y ( :,1), '— b’, 'LineWidth', 2) ; 

34 xlabel('x') ; ylabel('y(x) ') ; 

35 title ([' \sigma = ' num2str(sigma)]) ; 

36 sigma = 80 ; 

37 [t,y] = ode23s(Ovdpsig, tspan, yO); 

38 subplot (3, 2, 6) ; 

39 plot(t,y ( :,1), '— b', 'LineWidth', 2) ; 

40 xlabel('x') ; ylabel('y(x) ') ; 

41 title ([' \sigma = ' num2str(sigma)]) ; 

By executing the above MATLAB code, we get the Solutions for different 
values of a as shown in Figure 7.8. 
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FIGURE 7.8: Solution of the Vanderpool System for ct = 1,5,10,20,40 and 80. 


7.5 Python Solvers for IVPs 

Python has many means to solve IVPs of ODEs. Among the available tools 
are the scipy. integrate. odeint function and the gekko package. 

7.5.1 Solving ODEs with odeint 

The odeint function receives a vector containing the RHS of the ODEs Sys¬ 
tem, the vector of initial conditions and the time span. It returns the solution 
of the ODEs System. 
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Example 7.8 In this example, a python code LogmodWithPergrow.py will 
be used to solve the differential equation: 

x{t) = 4:Sin{2t)x{t) — x(0)=0.1, te [0,20] 

The Python code is: 


1 import numpy as np 

2 from scipy.integrate import odeint 

3 

4 f = lambda x, t: (4* (np.sin(2*t))*x-3*x**2) 

5 t = np.linspace{0.0, 20,0, 401) 

6 xO = 0.1 

7 

8 X = odeint(f, xO, t) 

9 

10 import matplotlib.pyplot as plt 

11 plt.figure(1, figsize=(10, 10)) 

12 plt.plot(t, X, color= 'orangered’ , Iw = 2) 

13 plt. grid (True, Is = ':') 

14 plt.xlabel ( ’ t' , fontweight= 'bold' ) 

15 plt.ylabel( 'X(t)' , fontweight= ’bold' ) 

16 plt.xticks(np.arange(0, 22, 2), fontweight= 'bold' ) 

17 plt.yticks(np.arange(0.0, 1.0, 0.1), fontweight= ’bold' ) 


Executing the code, shows in Figiire 7.9: 



FIGURE 7.9: Solution of the initial value problem using the odeint function. 
A Python code for solving the Vanderpole oscillator problem is as follows: 


1 import numpy as np 

2 from scipy.integrate import odeint 

3 import matplotlib.pyplot as plt 

4 global sigma ; 

5 tspan = [0, 200] ; yO = [2, 1] ; 

6 t = np.linspace(0.0, 200.0, 10001) 

7 vdpsig = lambda z, t: [z[l], sigma*(1-z[0]**2)*z[1]-z[0]] 

8 plt.figure(1, figsize = (20, 10)) ; 







Python Solvers for IVPs 


9 sigma = 1 ; 

10 y = odeint(vdpsig, yO, t) 

11 yl= y [ : , 0] 

12 plt. subplot (3, 2, 1) ; 

13 plt.plot(t,yl,color = 'crimson’, Iw = 2, label = r'$\sigma 

$' +str (sigma)) 

14 plt.xlabel ( 'X' , fontweight= 'bold’ ) ; plt.ylabel( 'y(x) ' , .. 

fontweight= 'bold' ) 

15 plt. legend () 

16 plt.grid(True, ls=':') 

17 

18 plt.xticks(np.arange (0,0, 220.0, 20), fontweight= 'bold' ) 

19 plt.yticks(np.arange(-2, 3), fontweight= 'bold’ ) 

20 

21 sigma = 5 ; 

22 y = odeint(vdpsig, yO, t) 

23 yl= y [ :, 0 ] 

24 plt.subplot(3, 2, 2) ; 

25 plt.plot(t,yl,color = 'crimson’, Iw = 2, label = r'$\sigma 

$' +str (sigma)) 

26 plt.xlabel( 'X' , fontweight= 'bold’ ) ; plt.ylabel ('y(x)' , .. 

fontweight= 'bold' ) 

27 plt. legend () 

28 plt.grid(True, ls=':') 

29 plt.xticks(np.arange(0,0, 220.0, 20), fontweight= 'bold' ) 

30 plt.yticks(np.arange(-2, 3), fontweight= 'bold' ) 

31 

32 sigma = 10 ; 

33 y = odeint(vdpsig, yO, t) 

34 yl= y [ : , 0] 

35 plt. subplot (3, 2, 3) ; 

36 plt.plot(t,yl,color = 'crimson', Iw = 2, label = r'$\sigma 

$' +str (sigma)) 

37 plt.xlabel (' X' , fontweight= 'bold' ) ; plt.ylabel (’y(x)' , .. 

fontweight= 'bold' ) 

38 plt. legend () 

39 plt.grid(True, ls=':') 

40 plt.xticks(np.arange (0,0, 220.0, 20), fontweight= 'bold' ) 

41 plt.yticks(np.arange(-2, 3), fontweight= 'bold' ) 

42 

43 sigma = 20 ; 

44 y = odeint(vdpsig, yO, t) 

45 yl= y [ :, 0 ] 

46 plt.subplot(3, 2, 4) ; 

47 plt.plot(t,yl,color = 'crimson', Iw = 2, label = r'$\sigma 

$' +str (sigma)) 

48 plt.xlabel (' X' , fontweight= 'bold’ ) ; plt.ylabel ('y(x)' , .. 

fontweight= 'bold' ) 

49 plt. legend () 

50 plt.grid(True, ls=':') 

51 plt.xticks(np.arange (0,0, 220.0, 20), fontweight= 'bold' ) 

52 plt.yticks(np.arange(-2, 3), fontweight= 'bold' ) 

53 

54 sigma = 40 ; 

55 y = odeint(vdpsig, yO, t) 

56 yl= y [ :, 0 ] 

57 plt.subplot(3, 2, 5) ; 
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58 plt.plot(t/yl,color = 'crimson', Iw = 2 , label = r'$\sigma = ... 

$' +str (sigma)) 

59 plt.xlabel( 'X' , fontweight= 'bold’ ) ; plt.ylabel( ’y(x)' , ... 

fontweight= 'bold' ) 

60 plt. legend () 

61 plt.grid(True, ls=':') 

62 plt.xticks(np.arange(0,0, 220.0, 20), fontweight= 'bold' ) 

63 plt.yticks(np.arange(-2, 3), fontweight= 'bold' ) 

64 

65 sigma = 80 ; 

66 y = odeint(vdpsig, yO, t) 

67 yl= y [ :, 0] 

68 plt. subplot (3, 2, 6) ; 

69 plt.plot(t,yl,color = 'crimson', Iw = 2, label = r'$\sigma = ... 

$' +str (sigma)) 

70 plt.xlabel( 'x' , fontweight= 'bold' ) ; plt.ylabel ( ’y(x) ' , ... 

fontweight= 'bold' ) 

71 plt. legend () 

72 plt.grid(True, ls=':') 

73 plt.xticks(np.arange (0,0, 220.0, 20), fontweight= 'bold' ) 

74 plt.yticks(np.arange(-2, 3), fontweight= 'bold' ) 


The Solutions of the System for different values of a are explained by Figure 
7.10. 
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FIGURET.10: Solution of the Vanderpool System for cr = 1,5,10,20,40 and 80. 
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7.5.2 Solving ODEs with Gekko 

GEKKO is a python optimization suite that is originally designed for solving 
dynamic optimization problems for mixed-integer, nonlinear, and differential 
algebraic equations (DAE) problems [2]. The default solver of Gekko is the 
IPOPT solver which is based on the interior point method. It includes other 
solvers such as APOPT, BPOPT, SNOPT, and MINOS [2]. 

The discretization of state equations is performed by using orthogonal col- 
location on finite elements, where in each finite element orthogonal collocation 
approximates control and state variables with polynomials [2]. 

To initialize a gekko model m in python, the following two instructions are 
used 


1 from gekko import GEKKO 

2 m = GEKKO(remote = False) 

Assigning the value False to the parameter remote informs gekko to set 
the server to be the local machine. 

If the model m contains a variable x that is bounded from below by Xmini 
from above by Xmax and has an initial value xq, then it can be declared as 
follows: 

X = m.Var(value = xo,lb = Xmin,ub = x^ax) 

Example 7.9 This example uses gekko to solve the SIR model: 

S{t) = A-aS{t)I{t)-nS{t),S{0) = So,te[{),T] 
i{t) = a.5(t)/(t)-(Ai + ^)/(t),/(0)=/o,tG [0,T] 

R{t) = /3/(t)-/zR(t),i?(0) = i?o,ie [0,T] 

with A = 0.05, /r = 0.05, q: = 2, /3 = 0.6 and T = 25. 

The gekko Python code is: 


1 #SolveSIRgekko .py 

2 from gekko import GEKKO 

3 import numpy as np 

4 import matplotlib.pYplot as plt 

5 

6 Lambda = 0.05 ; mu = 0,05 ; alpha = 2 ; beta = 0.6 

7 to = 0 

8 T = 25 

9 N = 1000 

10 m = GEKKO(remote=False) 

11 m.time = np.linspace(t0, T, N+1) 

12 S, I, R = m.Var{value=0,9), m.Var(value=0.1), m.Var(value=0.0) 

13 # Equations 

14 m.Equation(S.dt() == Lambda-alpha*S*I-mu*S) 

15 m.Equation(I.dt() == alpha*S*I-(mu+beta)*I) 

16 m.Equation(R.dt() == beta*I-mu*R) 

17 

18 m.options.IMODE = 4 # simulation mode 

19 m. solve () 
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20 t = m.time 

21 

22 plt.figure(1, figsize=(8, 8)) 

23 

24 plt.plot(t, S, color= ’darkblue’ , Is = Iw = 3, ... 

label=' Suceptible Population (S (t)) ' ) 

25 plt.plot(t/ I, color= 'crimson' , Is = Iw = 3, ... 

label= ’Infected Population (I(t))') 

26 plt.plot(t, R, color= 'darkmagenta ' , Is = ': ' , Iw = 3, ... 

label= ’Recovered Population (R(t))') 

27 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

28 plt.ylabel (' Population' , fontweight= 'bold' ) 

29 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

30 plt.yticks(np.arange(0, 1.1, 0.1), fontweight= ’bold' ) 

31 plt.grid(True, ls='— ') 

32 plt.axis([0,0, T, 0.0, 1.1]) 

Executing the code SolveSIRgekko. py 

runfile(’/media/WindowsDatal/PyFiles/SolveSIRjekko.py’, 
wdir=’/media/WindowsDatal/PyFiles’) 


APMonitor, Version 0.8.9 
APMonitor Optimization Suite 


- APM Model Size 

Each time step contains 


Objects : 0 
Constants : 0 
Variables : 3 
Intermediates; 0 
Connections ; 0 
Equations ; 3 
Residuals ; 3 


Number of state variables: 
Number of total equations: - 
Number of slack variables: - 


1200 

1200 

0 


Degrees of freedom : 0 

solver 3 not supported 

using default solver: APOPT 

Dynamic Simulation with APOPT Solver 


Iter Objective Convergence 
0 2.90508E-13 1.15000E-01 
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1 1.50866E-13 

2 4.75681E-15 

3 1.74925E-15 

4 1.39990E-13 

5 6.11632E-16 

6 1.54265E-16 

7 3.91236E-17 

8 9.97549E-18 

9 2.46346E-18 


1.14988E-01 
3.14821E-01 
3.13396E-01 
2.80934E+01 
2.59675E+00 
2.30466E+00 
1.83967E+00 
9.28526E-01 
6.43451E-01 


Iter Objective Convergence 


10 5.36666E-19 

11 1.04749E-19 

12 2.01098E-20 

13 1.59276E-21 

14 6.84198E-24 

15 6.43007E-29 

16 6.43007E-29 


3.23197E-01 
8.50434E-02 
1.72751E-02 
3.67618E-03 
5.41524E-04 
5.76856E-06 
5.76856E-06 


Successful solutiori 


Solver ; IPOPT (v3.12) 

Solution time ; 6.480000000010477E-002 sec 

Objective ; 0.OOOOOOOOOOOOOOOE+000 

Successful solution 


The solution graph is shown in Figure 7.11. 



FIGURE 7.11: Solution of the SIR model in [0,10] using the gekko package. 
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Example 7.10 In this example, gekko will be used to solve the stiff system 
of differential equations: 


dyijt) 

dt 

dyijt) 

dt 

for 0 < t < 5. 

The code is: 


-1002 * 2/1 (t) +1000* 2/i (t), 2/i(0) = l 
yi{t) + y2*{^ + y2{t)), 2/2(0) = ! 


1 from gekko import GEKKO 

2 import numpy as np 

3 import matplotlib.pYplot as plt 

4 

5 tO = 0 

6 T = 5 

7 N = 2000 

8 m = GEKKO(remote=False) 

9 m.time = np.linspace(tO, T, N+1) 

10 yl, y2 = m.Var{value=l.0), m.Var(value=2.0) 

11 t = m.time 

12 # Equations 

13 m.Equation(yl.dt() == -1002*yl+1000*y2**2) 

14 m.Equation(y2.dt{) == yl-y2*{l+y2)) 

15 

16 m.options.IMODE = 4 # simulation mode 

17 m. solve () 

18 



0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 


FIGURE 7.12: Solution of the stiff system of ODEs in [0,5] using the gekko 
package. 
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19 plt. f igure (1, figsize=(8, 8)) 

20 

21 plt.plot(t, yl, color= ' darkblue' , Is = Iw = 3, label= ' y 1 (t) ' ) 

22 plt.plot(t/ y2, color= 'crimson Is = ' — Iw = 3, label= 'y2 (t) ' ) 

23 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

24 plt.ylabel( 'y(t)' , fontweight= ’bold' ) 

25 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

26 mxS = np.ceil (max(max (y1), max(y2))) 

27 plt.yticks (np.arange(0, mxS+mxS/10 , mxS/10 ), fontweight= 'bold’ ) 

28 plt.grid(True, ls='— ') 

29 plt.axis([0,0, T, 0.0, mxS+mxS/10]) 

30 plt. legend () 


Executing the code SolveSIRgekko. py shows the solution graph in Figure 
7.12. 
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Nonstandard Finite Difference Methods 
for Solving ODEs 


Abstract 

Standard numerical methods are initially designed to solve a class of gen- 
eral problems without considering the structure of any individual problems. 
Hence, they seldom produce reliable numerical Solutions to problems with 
complex structures such as nonlinearity, stiffness, singular perturbations and 
high oscillations. While the explicit difference schemes can solve such prob¬ 
lems with low computational cost, they suffer the problems of small stability 
regions and hence they suffer severe restrictions on step sizes to achieve con¬ 
venient results. On the other-hand, the implicit finite difference schemes enjoy 
wide stability regions but suffer high associated computational costs and their 
convergence orders cannot exceed one order above the explicit methods for the 
same number of stages [1]. 

This chapter highlights some of the cases in which the Standard finite 
difference schemes fail to find reliable Solutions, then it discusses the rules upon 
which the nonstandard schemes stand with several examples to explain the 
idea. MATLAB® and Python are used to implement the solution algorithms 
in all the sections. 

The chapter is organized as follows. The first section discusses some numer¬ 
ical cases in which the Standard finite difference methods give inappropriate 
Solutions. In the second section, the construction rules of nonstandard finite 
difference methods are introduced. Exact finite difference schemes based on 
nonstandard methods are presented in Section 3, for solving some given ini- 
tial value problems. Finally, in the fourth section, design of nonstandard finite 
difference schemes -for the case when exact finite differences are hard to find¬ 
is presented. 


8.1 Deficiencies with Standard Finite Difference 
Schemes 

The major problem that encounters the Standard finite difference methods 
is the numerical instability [38, 40]. A discrete model of a differential 
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equation is said to have numerical instability if there exist Solutions to the 
finite difference equation that do not correspond to any of the possible Solu¬ 
tions of the differential equations. Mickens pointed to some causes of numerical 
instability [37]. These causes are: 


(i) 


Representing a derivative in the differential equation with a discrete deriva¬ 
tive of different order. An example to this situation is when representing 
the logistic model 


— =P(l-P),P(0)=Po 


using the centeral difference scheme: 


pj+i _ pi-i 


pj(l - PJ) =» PJ+I = pJ-i -p2/iPJ(1 - P^). 


Figure 8.1 shows the solution of the logistic model P{f) = P(t)(l — 
P{t)),P{0) = 0.5 using the centeral, forward and backward difference 
schemes. The figure show that the Solutions obtained by both the for¬ 
ward and backward difference scheme have the same behavior as the 
original model, where as the centeral finite difference scheme is irrelative 
to the exact solution. 


(ii) Using the Standard denominator function h [40]. The first and second 
derivatives at a point tj are approximated by: 

du ^ u{U+i)-u{ti) , ^ u{ti)-u{U-i) 

dt ^ h ^ ' dt ^ h 

d?u^^^ _ u{tij^i)-2u{ti)+u{ti+i) 

dP^"’ 

In many cases, selection of the classical denominator function is not 
suitable and can lead to numerical instability. To see such an example, 
consider the first order IVP: 

^^^=Ay(f)-ht,y(0) = 0. 

The exact solution of this IVP is 

2/W = ^ . 

The interval [0,1] will be divided into 1000 subintervals, by points 0 = 
to < ti < ... < tiooo = 1 and each subinterval has a length h = 0.001. The 
forward Euler’s method, Implicit trapezoidal rule and classical fourth- 
order Runge-Kutta method will be used for solving the given problem 
at the discrete points for different values of parameter A. Theoretically, 
the error of Euler’s method shall be of 0{h) = 0(10“^), the error of the 
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FIGURE 8.1: Solution of the logistic model P{t) = P{t){l — P{t)),P{0) = 0.5 
using the forward, backward and Central finite difference schemes. 


implicit trapezoidal rule shall be of = 0(10 and the error of 

the fourth-order Runge-Kutta method shall be of 0{h'^) = 0(10“^^). 

The Python code to compute the errors of the three methods for different 
values of A is: 


1 import numpy as np 

2 import matplotlib.pyplot as plt 

3 

4 f = lambda s, v, B: B*v+s 

5 N = 1000 

6 h, t = 1.0/N, np.linspace(0, 1, N+1) 

7 yO = 0 

8 MaxRows = 12 

9 Eui = np.zeros((MaxRows, len(t)), float) 

10 rk.4 = np. zeros ( (MaxRows / len(t)), float) 

11 itr = np.zeros((MaxRows, len(t)), float) 

12 EulError = np.zeros((MaxRows, ), float) 

13 rk.4Error = np. zeros ( (MaxRows, ), float) 

14 itrError = np.zeros((MaxRows, ), float) 

15 A = np.zeros((MaxRows,), float) 
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16 row = 0 

17 A[row] = 5,0 

18 i = 0 

19 while row < MaxRows: 

20 # y is the exact solution of the problem 

21 y = 1/A[row](np.exp(A[row]*t)-(A[row]*t+l)) 

22 plt.plot{t, y, lw=2, label='A = ' +str (A[row])) 

23 plt. legend () 

24 Eui [i, 0] = yO 

25 rk4 [i, 0] = yO 

26 itr[i, 0] = yO 

27 for j in range (N): 

28 # Solving with Euler's method 

29 Eul[i, j+1] = Eul[i, j] + h*f(t[j], Eul[i, j], A[row]) 

30 # Solving with implicit trapezoidal rule 

31 kl = f(t[j], rk4[i, j], A[row]) 

32 k2 = f(t[j]+h, itr[i, j]+h*kl, A[row]) 

33 itr[i, j+1] = itr[i, j] + h/2*(kl+k2) 

34 # Solving with the classical fourth-order Runge-Kutta ... 

method 

35 kl = f(t[j], rk4[i, j], A[row]) 

36 k2 = f(t[j]+h/2, rk4[i, j]+h/2*kl, A[row]) 

37 k3 = f(t[j]+h/2, rk4[i, j]+h/2*k2, A[row]) 

38 k4 = f(t[j]+h, rk4[i, j]+h*k3, A[row]) 

39 rk4[i, j+1] = rk4[i, j] +h/6*(kl+2*k2+2*k3+k4) 

40 # computing the norm-infinity error for the three methods 

41 EulError[i] = np.linalg.norm(y-Eul[i, :], np.inf) 

42 itrError[i] = np.linalg.norm(y-itr[i, :], np.inf) 

43 rk4Error[i] = np.linalg.norm(y-rk4[i, :], np.inf) 

44 i += 1 

45 row += 1 

46 if row > MaxRows: 

47 break 

48 else: 

49 A[row] = A[row-l] + 2.0 

50 print ( '-' ) 

51 print ( ' A\t Euler Error\t\t Imp. Trapz Err\t\t RK4 Error') 

52 print ( ' - ' ) 

53 for row in range (MaxRows): 

54 print ( '{0: 4 . Of }' . format (A[row]), ... 

'\t' , '{0:1.8e}' . format (EulError[row]), \ 

55 ' \t ’ f '{0:1.8e} format (itrError[row]), '\t', ... 

'(0:1.8e}' , format (rk4Error[row])) 

56 print ( ' - ' ) 


Running the code gives the following results: 

runfile(’/media/WindowsDatal/PyFiles/ 
numinstduetotrivstepfun.py’, 
wdir=’/media/WindowsDatal/PyFiles’) 


A Euler Error Imp. Trapz Err RK4 Error 


1 1.35789622e-03 3.56321584e-07 2.30926389e-14 
3 1.00002555e-02 5.19127795e-06 4.50417481e-12 
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5 

7.35013397e-02 

4.52531609e-05 

1.53952406e-10 

7 

5.39170234e-01 

3.52721966e-04 

3.11633030e-09 

9 

3.94740396e+00 

2.65343074e-03 

4.88584107e-08 

11 

2.88444591e+01 

1.97153658e-02 

6.58044428e-07 

13 

2.10373074e+02 

1.45836606e-01 

8.01259330e-06 

15 

1.53146894e+03 

1.07702617e+00 

9.07992144e-05 

17 

1.11283670e+04 

7.94936660e+00 

9.75035611e-04 

19 

8.07191089e+04 

5.86610000e+01 

1.00415309e-02 

21 

5.84468245e+05 

4.32849955e+02 

1.00014394e-01 

23 

4.22478467e+06 

3.19388027e+03 

9.69289817e-01 

25 

3.04879507e+07 

2.35668600e+04 

9.18238944e+00 

27 

2.19661624e+08 

1.73896191e+05 

8.53281952e+01 

29 

1.58017839e+09 

1.28317310e+06 

7.79939406e+02 


It could be seen that the errors of the three methods are increasing as 
A increases, although the step-size h is kept constant. 

(iii) using local approximations of nonlinear terms [40]: The Standard finite 
difference schemes represent nonlinear terms such as x‘^{tj) using a local 
approximations such as or Mickens showed that such local 

approximations of nonlinear terms could lead to numerical instability. 
An example given by Mickens is the solution of the exponential decay 
model 

^ = -Pit), P(0) = 0.5. 

The forward Euler’s discretization of the exponential decay model is 
given by: 


pn+l ^pn_ J^pn ^ q _ j^^pn ^ pO 

For different values of the step size h, the resulting Solutions are shown 
in Figure 8.2. 

The Solutions obtained for the exponential decay model include solution 
profiles with asymptotic stable, periodic and unstable dynamics, where 
the periodic and unstable Solutions do not correspond any true solution 
of the exponential decay model. The first graph alone, where the popu- 
lation size decays to zero agrees with the true solution of the differential 
equation. 

Another example is the use of the Heun’s method to solve the logistic 
model 

dx{t) 


dt 


x{t){l — x{t)), a:(0) = 0.5 
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Time {t) Time {t) 




FIGURE 8.2: Numerical solution of the exponential decay model based on 
forward Euler’s method with different step-sizes. 


The Python code is: 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 
23 


import numpy as np 
import matplotlib.pyplot as plt 
steps = [0,5, 2.75, 3.0, 3.35] 
plt.figure(1, figsize=(12, 12)) 
T = 100 

for k in range(4) : 
h = steps[k] 

N = round(T/h) 
t = np.linspace(0, T, N+1) 

P = np.zeros-like(t) 

P[0] =0.5 


for j in range (N) : 

kl = P[j] * (1.0-P[j]) 

Pest = P[j]+h*kl 
k2 = Pest* (1.0-Pest) 

P[j+1] = P[j] + h/2*{kl+k2) 
plt. subplot{2, 2, k+1) 

plt.plot(t, P, color = 'blue', lw=3, label = 'h = ... 
’+str (h)) 

plt.xlabel (' Time (t) ', fontweight= ' bold ' ) 

plt.ylabel{ ' Population {P(t))', fontweight = 'bold') 

plt.legend() 

plt.grid(True, Is = '--') 

plt.xticks{np.arange(0, 105, 10,0), fontweight = 'bold') 
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FIGURE 8.3: Solution of the logistic model using Heun’s method for different 
values of step-size. 


24 mnp, mxp = np.floor( 10*min (P))/10, np.ceil(1 0*max (P))/10 

25 plt.yticks(np.arange(mnp, mxp+(mxp-mnp)/10, ... 

(mxp-mnp)/10), fontweight = 'bold') 


Figure 8.3 shows the solution of the logistic model with the Heun’s 
method for different values of the step-size h. 

From Figure 8.3, corresponding to some values of the step-size h, the 
numerical solution of the logistic model could have periodic and chaotic 
behaviors, which are not corresponding to any solution of the logistic 
model. 


8.2 Construction Rules of Nonstandard Finite 
Difference Schemes [38] 

The numerical stability that accompany the design of the Standard finite dif¬ 
ference schemes, show that due to some inherent errors in the Standard mod- 
els, it becomes very expensive to retrieve the true information, causing these 
Standard schemes to present unreliable results. 
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Considering the causes leading to numerical instabilities in the Standard 
finite difference schemes, four nonstandard construction rules have been estab- 
lished by Micken’s for the design of such numerical scheme. These rules are 
stated in [37]: 

(I) Because using a discrete derivative of order that differs than the order 
of the differential equation can lead to numerical instability, the order of 
the discrete modei shall be as same as the order of the differential equation. 
Under this rule the Central finite difference scheme cannot be used as 
an approximation of the first derivative in the discrete modei of a first- 
order differential equation. Either the forward or backward difference 
schemes can be used. 

(II) Nonstandard denominator functions have to be used for the discrete repre- 
sentation of the continuous derivative. A nonstandard discrete represen- 
tation of ^ at t = tj is of the form: 

dy _ y(tj+i)-il}{\,h)y{tj) 

dt (j){\,h) 

where A is a vector of the modePs parameters, h = tj^i — tj. The numer¬ 
ator and denominator functions h) and ^l'(A, h) shall fulfill the prop- 
erties: 

h) ^ 1 and (/)(A, /i) —>■ /i as /i —>■ 0 

An example to show how the use of nontrivial denominator can lead to 
numerical instability, consider the exponentially decay modei 

^ = -P(t),P(0) = 0.5. 

In Euler’s method, suppose that the trivial denominator function h is 
replaced by the denominator function 

m = —, 

hence the resulting discrete modei is 

pj+i = pO = 0.5 

The Python code expdecnsden.py is used to solve the exponential 
decay modei for different values of the step-size h: 


1 #expdecnsden.py 

2 import numpy as np 

3 import matplotlib.pyplot as plt 

4 steps = [ 0 . 5 , 1 . 05 , 2 . 5 , 5 , 0 ] 

5 plt.figure(1, figsize={12, 12)) 
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6 T = 105 

7 for k in range(4) : 

8 h = steps[k] 

9 phi = (1-np,exp(-h))/h 

10 N = round(T/h) 

11 t = np.linspace(0, T, N+1) 

12 P = np . zeros-like (t) 

13 P [ 0 ] = 0.5 

14 for j in range (N): 

15 P[j + 1] = (1.0-phi) *P [ j] 

16 plt.subplot(2, 2, k+1) 

17 plt.plot(t, P, color = 'blue', lw=2, label = ’h = ... 

' +str (h)) 

18 plt.xlabel( ' Time (t)', f ontweight= ’ bold ' ) 

19 plt.ylabel( ' Population (P(t))’, fontweight = 'bold') 

20 plt. legend () 

21 plt. grid (True, Is = 

22 plt.xticks(np.arange(0, 110, 15), fontweight = 'bold') 

23 mnp, mxp = np.floor(1 0*min (P))/10, np.ceil( 10*max (P))/10 

24 plt.yticks(np.arange(mnp, mxp+(mxp-mnp)/10, ... 

(mxp-mnp)/10), fontweight = 'bold') 


The resuit of executing the code is Figure 8.4, in which the exponential 
decay model is solved with a nonstandard denominator function for 
different values of the step size h. 




0 15 30 45 60 75 90 105 

Time (t) 



Time (t) 

0.50 
0.45 
0.40 
:: 0.35 
w 0.30 
0.25 
3 0.20 
0- 0.15 
0.10 
0.05 
0.00 

0 15 30 45 60 75 90 105 

Time (t) 



FIGURE 8.4: Solution of the decay model using a nonstandard denominator 
function, for different values of the step-size. 
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(III) Nonlinear terms shall be approximated by using non-local approximations: 
terms like y^{t) shall be represented by or instead of 

where y^ ~ y{tj). For example, the discrete model corresponding 
to the logistic model 

^=y(t)(l-y(t)),y(0) = 0.5 

could be 


y3 + l-y] 


J(l_yj+1) ^ = 0.5 


h y ^ l + hyi 

A Python code nonlocapplog.py is used to solve the logistic model 
using nonlocal approximations. The code is 


1 

# nonlocapplog.py 


2 

import numpy as np 


3 

import matplotlib.pyplot as plt 


4 

steps = [0.75, 3.75, 7.5, 15.0] 


5 

plt.figure (1, figsize={12, 12)) 


6 

T = 75 


7 

for k in range(4) : 


8 

h = steps[k] 


9 

N = round.(T/h) 


10 

t = np.linspace(0, T, N+1) 


11 

P = np.zeros-like(t) 


12 

P[0] = 0.5 


13 

for j in range (N) : 


14 

P[j + 1] = (1.0+h)*P[j]/(l+h*P[j]) 


15 

plt.subplot (2, 2, k+1) 


16 

plt.plot(t, P, color = 'blue', lw=3, label = 
'+str (h)) 

'h = ... 

17 

plt.xlabel (' Time (t)’, f ontweight= ' bold ' ) 


18 

plt.ylabel( ' Population (P(t))', fontweight = 

'bold' ) 

19 

plt.legend() 


20 

plt.grid(True, Is = 


21 

plt.xticks(np.arange(0, 80, 7.5), fontweight 

= 'bold' ) 

22 

mnp, mxp = np.floor(1 0*min (P))/10, np.ceil( 10*max (P))/10 

23 

plt.yticks(np.arange(mnp, mxp+(mxp-mnp)/10, 
(mxp-mnp)/10), fontweight = 'bold') 



Figure 8.5 is obtained by executing the python code. It shows the Solu¬ 
tions of the logistic model using different values of step-sizes. AU these 
Solutions correspond the true solution of the differential equation model. 
Hence, for all values of step-sizes, the discrete model does not suffer 
numerically instability. 

(IV) Dynamical consistency between the solutioris of the discrete model and 
differential equation. That means all the properties possessed by the 
solution of the differential equation must be possed by the solution of the 
discrete model. Particular properties include positiveness, monotonicity, 
boundness, limit cycles and other periodic Solutions, etc. 
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FIGURE 8.5: Solution of the logistic model using a non-local approximation 
of y^{t) with different values of step-sizes. 


A nonstandard finite difference scheme is a discrete representation of the 
differential equation which is constructed based on the nonstandard rules. 


8.3 Exact Finite Difference Schemes 

Given an initial value problem 

^ = y{a) = y°,t&[a,b] ( 8 . 1 ) 

where A is the set of model parameters. Suppose that the solution y{t) of the 
given initial value problem is 

y{t) = F{t,y{t),y°,X). (8.2) 

Let be a positive integer, h = and tj = a + jh,j = 0,...,N. Gonsider 

a finite difference scheme 


= g{h,tj,y^ ,X) 


( 8 . 3 ) 
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and « y{tj)- Suppose that the solution of the discrete finite difference 
scheme is 

(8.4) 

According to Mickens [37] Equation (8.3) is an exact finite difference 
scheme of the differential equation (8.1) if its solution (8.4) is as same as the 
solution of associated differential equation (8.2). That means 

yO =G{tj,h,y^ ,y°,X) = F{tj,y{tj),y°,X) =y{tj). 

Mickens [39] described a mechanism to construet exact finite difference 
schemes for many kinds of models. The idea is to incorporate some qualitative 
features of the solution of the problem into the discrete model. Qualitative 
features can be obtained from the linear part in the model. The nonlinear 
parts are dealt with using the nonlocal approximation techniques. 


8.3.1 Exact Finite Difference Schemes for Homogeneons 
Linear ODEs 

In this section, exact finite difference schemes will be derived for three kinds of 
linear and homogeneous ODEs. They include linear first order, linear second 
order equation and linear system of first order ODEs. 


8.3.1.1 Exact Finite Difference Schemes for a Linear Homogeneous 
First-Order ODE 


Given the initial value problem: 


^^^ = -ay(f),y(0) = y° 

The exact solution of the given IVP is y{t) = The bases solution 

function is yi{t) = e““*. To determine the exact finite difference scheme of 
the given IVP, consider the determinant: 


y^ 

e “L 

= 

y^ 

1 

yj + l 

g—atj + i 


yJ+1 

e-ah 


= 0 = e °‘^y^ 


Subtracting y^ from the two sides and multiplying the right-hand side by a 
and dividing by a gives the discrete scheme: 


7 + 1 7 

yj^ _yJ — _Q, 


1 — e 


— ah 


yj + l-yj 

l-g-ah 


= —ay-' 


from which, 
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Hence, instead of using the Standard denominator function h, using the denom- 

1_ —ah 

inator function — - - will resuit in an exact finite difference scheme. 

Q; 

Example 8.1 (First order linear ODE [39]) In this example, an exact 
finite difference scheme will be derived for the exponential decay model 


dyjt) 

dt 


TT , , , , 1 

= 2 


The following Python code solves the exponential decay model using the 
exact finite difference scheme with different values of the step size h and shows 
the corresponding infinity norm of the difference between the exact solution 
and the solution of the discrete model. 


1 

import numpy as np 


2 

import matplotlib.pYplot as plt 


3 

plt.figure(1, figsize=(12, 12)) 


4 

alpha = np.pi/4 


5 

T = 100 


6 

steps = np.array([0.1, 0.5, 1.0, 2.0, 
12.5, 20.0, 25.0, 50.0, 100.0]) 

4.0, 5.0, 6.25, 10.0, ... 

7 

Errors = [] 


8 

for k in range(l, len(steps)): 


9 

h = steps[k] 


10 

phi = (1-np.exp(-alpha*h))/alpha 


11 

N = int(round (T/h)) 


12 

t = np.linspace(0, T, N+1) 


13 

P, PEx = np.zeros-like(t), np.zeros.like(t) 

14 

p[0] = 0.5 


15 

PEx[0] = 0.5 


16 

for j in range (N): 


17 

P[j+1] = (1.0-phi*alpha)*P [ j] 


18 

PEx[j + l] = P [ 0]*np.exp{-alpha 

*t [ j + 1]) 

19 

MaxError = np.linalg.norm(PEx-P, 

np.inf) 

20 

Errors.append([h, MaxError]) 


21 

print ( ' - ' ) 


22 

print ( ’ h\t\t Error') 


23 

print ( '-' ) 


24 

for j in range ( len (Errors)): 


25 

print ('{0=3-2f}' . format (Errors[j][0]) 

'{0 : 1.8e} ' . format {Errors[j][1])) 

+ '\t' + ... 

26 

print ( ' - ' ) 



Executing the code gives the following results: 


h 


Error 


0.50 2.77555756e-17 
1.00 4.16333634e-17 
2.00 8.67361738e-19 
4.00 7.63278329e-17 
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5.00 5.20417043e-18 

6.25 1.30104261e-17 

10.00 4.24465151e-17 

12.50 3.25836634e-17 

20.00 1.24790452e-18 

25.00 2.25704533e-17 

50.00 4.40824356e-18 

100.00 3.88652225e-35 


8.3.1.2 Exact Finite Difference Scheme for Linear Homogeneous 
Second Order ODE 


Example 8.2 (second order linear ODE [39]) In this example, the exact 
finite difference scheme will be established for solving the second order BVP: 


d?u 


{f)+uj‘^u{f) 


0,it(0) 


WQ, u{2) = Ui 


The characteristic equation corresponding to the given BVP is 


r^+o;^ = 0, 


whose Solutions are 

ri_2 = 'Fuji. 

Hence, the bases solution functions of the BVP are 

ui{t) = sin (ujt) and U 2 ft) = cos (ujt). 
Consider the determinant: 


= 0 . 


Using the trigonometric identity sin(a) cos(6) — cos(a) sin(&) =sm(a —6) and 
remembering that tj+i — tj = tj — tj-i = h,tj + 1 — tj-i = 2h the determinant 
is expanded into: 

sin {u}h)u^~^ — sin {2u!h)u^ + sin {ujh)u^'^^ = 0 => —2cos {ujh)u^ = 0 



cos(a;fj_i) 

sin (ujtj-i) 


cos (ujtj) 

sin {ujtj ) 

d+1 

cos{u}tj+i) 

sin 


Subtracting 2u^ from and adding 2cos{uih) to the two sides and using the 

ujh 
2 


trigonometric identity cos{uih) = 1 —2sin^(^) gives: 


'F ^ — 2u^ + = —4sin 


2 ujh 


■ ^ — 2u-’= — ^sin^ 
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Finally, dividing the two sides of equation by ^sin^ and adding 

to the two sides gives the exact finite difference scheme 


'F ^ — 2u^ + 


+ lJ^u^ = 0 


Hence, the exact finite difference scheme is obtained by selecting the denomi¬ 
nator function 


(j){uj,h) 


2 

— sin 
co 


Loh 


As /i —> 0, sin(^) —> and hence, 4){uj,h) —>• h. 

The exact solution of the problem is: 


'^exact{t) — 2 COS (wt) 


5 — |cos(2w) 
sin(2w) 


sin (ojt) 


To solve the given BVP using MATLAB for uq = 1.5 and ui = 0.5, the 
time space [0,2] is divided into A^-subintervals, where Ai is a positive integer. 
The solution is obtained by solving the linear system 



0 0 0 
0 0 0 
0 0 0 


0 

0 

0 


0 

0 

0 


0 

0 


0 0 0 
0 0 0 
0 0 0 



0 0 1 


r . 0 n 



U 


«0 



0 



0 



0 



0 

. . 


. “1. 


The MATLAB script SolveHarmOsc .m solves the BVP for uj = 16 and N = 
10,20,40 and 80: 


1 ciear ; clc ; clf ; 

2 a=0.0;b=2.0; 

3 w= 16 ; uO = 1.5; ul = 0.5 ; 

4 C1 = 1.5 ; C2 = {0.5-1, 5*cos {2*w)) /sin (2*w) ; 

5 figure(l) ; Error = [] ; 

6 N=10;k=l; 

7 while k < 4 

8 t = linspace(a, b, N+1) ; h = (b-a)/N ; 

9 phi = 2 /w*sin (w*h/2.0) ; 

10 A = . . . 

full (gallery { ' tridiag ' N+1,1/phi ^ 2 , w" 2-2/phi "2,1 /phi " 2) ) 

11 A(l, 1)=1.0; A(end, end) = 1,0 ; 

12 A(l, 2) = 0.0 ; A(end, end-1) = 0.0 ; 

13 g = zeros(N+l, 1) ; 

14 g(l) = uO ; g(end) = ul ; 

15 u = A\g ; 

ux = Cl*cos(w*t)’ + C2*sin(w*t)' ; 


16 
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17 Error = [Error; [N, norm(ux-u, inf) ] ] ; 

18 subplot(2, 2, k) ; 

19 plot (t, u, '-b', 'LineWidth', 2) ; 

20 xlabel ( ' t' ) ; 

21 ylabel ( ' u (t ) ' ) ; 

22 legend(['N = ' num2str (N)]) 

23 set (gea, ' fontweight ' , 'bold') ; 

24 set (gea, 'fontsize', 12) ; 

25 set (gea, 'XTick', a: (b-a)/10:b) ; 


26 

mnu = 

fleor (10*min (u))/10 ; 

: mxu = 

eeil (10*max 

27 

set (gea, 'YTiek', mnu:(mxu- 

-mnu)/10 

:mxu) ; 

28 

axis ( 

[a, b, mnu, mxu]) ; 



29 

grid ' 

en ; 



30 

N = 2 

*N ; 



31 

k = k 

+ 1 ; 



32 

end 




33 

fprintf ( ' 



-\n') ; 

34 

fprintf ( ' 

N\t\t Error\n') ; 



35 

fprintf ( ' 



-\n') ; 

36 

N = 100 ; 




37 

for j = 1 

; 4 



38 

fprintf ( ' %3i\t\t%l.12e\n' , 

Error(j 

, 1), Error 

39 

end 




40 

fprintf ( ' 



-\n') 


By exeeuting the MATLAB code, the Solutions of the BVP for the different 
values of N are shown in Figure 8.6 and the infinity norm errors are also shown. 


N Error 


10 3.330669073875e-15 
20 6.883382752676e-15 
40 4.996003610813e-15 
80 1.088018564133e-14 


If the Standard denominator function h is used instead of (j){uj,h), then the 
table of errors will be: 


N Error 


10 1.308207805131e+00 
20 3.921038479674e+00 
40 1.694674604840e+00 
80 5.894499384984e-01 
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t 



t 



FIGURE 8.6: Solutiori of the harmonic oscillator system using an exact finite 
difference scheme for different numbers of subintervals. 


8.3.1.3 Exact Finite Difference Scheme for a System of Two Linear 
ODEs 

Given a system of two ODEs: 

^ = ax{t) + by{t) 

^ = cx{t) + dy{t) 

which can be written in the matrix form: 
dz 


dt 


= Az(t) = 


a b 

x{t) 

c d 

. yl*). 


(8.5) 
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where 


z{t) 


x{f) 

y{t) 


and a,b,c and d are real numbers. 

Matrix A is similar to its Jordan form J G hence there exists an 

invertible matrix P G such that J = P~^AP. The diagonal elements 
of J are the eigenvalues of A and the columns of P are the eigenvectors or 
generalized eigenvectors of matrix A. The Jordan form J oi A has one of three 
forms: 


Ai 

0 


■ A 

0 ■ 

or 

■ A 

1 ■ 

0 

A 2 

5 

0 

A 

0 

A 


with Ai A 2 0 and A 0. 

In MATLAB, the Jordan form of a matrix A can be found by using the 
MATLAB function jordan, which returns the matrices P and J. 


» A = [2, 0, 0; 1, 1, -3; -1, -1, -1] ; 

>> [P, J] = jordan(A) ; 

>> disp(’P = ’) ; disp(P) ; 

P = 

0 0 l.OOOOe+00 

1.2500e-01 1.5000e+00 -1.2500e-01 

1.2500e-01 -5.0000e-01 -1.2500e-01 

>> disp(’J = ’) ; disp(J) ; 

J = 

-2 0 0 

0 2 1 

0 0 2 


In Python, the Jordan form of a matrix A can be found by using the 
method jordan_form of a symbolic matrix. 

In [1]: import numpy as np 
In [2]; import sympy as smp 

In [3]; A = np.arrayC[[2, 0, 0], [1, 1, -3], [-1, -1, -1]]) 

In [4]; P, J = (smp.Matrix(A)).jordan_form() 

In [5]: P, J = np.array(P), np.array(J) 

In [6]; print(’P = \n’, P) 

P = 

[[0 0 - 2 ] 

[1 -3 1] 

[110]] 

In [7]: print(’J = \n’, J) 

J = 

[[-2 0 0 ] 

[0 2 1 ] 

[0 0 2 ]] 
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Now, equation (8.5) can be written as 

dz 


dt 


= PJP-^Z. 


Multiplying the two sides of equation (8.5) by P~^ and using the linear trans- 
formation 

u{t) = P~^z{t) 

Equation (8.5) can be written as 



( 8 . 6 ) 


The solution of the linear system (8.5) is obtained by solving the linear 
System (8.6) and using the linear transformation 


z{t) = Pu{t). 


In [48], the exact finite difference schemes are derived for the three kinds 
of Jordan forms. The author proved that the linear system (8.6) has an exact 
finite difference scheme of the form: 




</> 




(8.7) 


where (j) and 9 are to be determined in terms of the step-size h and the eigen- 
values of A. The functions (j) and 9 are as follows: 

I. In the case that matrix A has two distinet roots Ai and A 2 , that is 


Ai 0 
0 A 2 ’ 


(f) and 9 are 


and 


(Ai-A2)(e^i^-I) 

A1A2 (e-''!^ — e'^ 2 ft-) 

A2(e^i^-I)-Ai 

(Ai-A2)(e^i'‘-l)(e^2'i-i) 


II. In the case that A has a repeated eigenvalue A and Dim( 2 l — XI) = 2, 
that is 


J = 


A 0 
0 A ’ 


then, 



-I 

-I) + A 
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Hence, if 6 is chosen to be then 

2(e^'^-l) 

X{e^h + iy 

and if 

^ 1 - + A/ie^^ 

U = - 7j -, 

(gA/i_l)2 

then 

III. In the case that A has a repeated eigenvalue A and Dim(A—AI) = 1, 
that is 

.[Ali 


0 A ’ 


then (j) and 9 will have the forms: 


1 - + Xhe^’^ 

(gAh_l)2 


(e^ 

Example 8.3 In this example, an exact finite difference scheme will be estab- 
lished to solve the linear system: 

x{t) = —2a;(t)+ y(t),a;(0) = 1, 

ijff) = a;(t)-y(t),y(0) = 0.5 

To solve this linear system of equations, let z{f) = [x(t),j/(t)]^, then the 
matrix form of the system is: 


z{t)= I z{t),z{Q)= 


The eigenvalues of the coefficient matrix are: 

, 3 ^/5 

Ai,2 = -2T^ 

and the corresponding eigenvectors are: 


■«1,2 = 
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The matrices J (Jordan form) and P are: 

0 


J = 


3 _ 

2 2 


0 


3 I V5 
2 2 


and P = 


1 , 

1 


The exact solution of the linear system is: 

jAlt 0 


zit)=P 


Q ^lambda2t p—1 


1_ \/5 


The MATLAB script SolveLinSysExact .m implements the exact finite 
difference scheme for solving the linear system, and shows the approximation 
errors at the discrete points: 


1 %SolveLinSysExact .m 

2 ciear ; clc ; clf ; 

3 a= 0.0 ;b= 10.0; 

4 A = [-2, 1; 1, -1] ; 

5 N = 40 ; 

6 t = linspace(a, b, N+1) ; 

7 h = (b-a)/N ; 

8 [phi, tht, P, J] = CompPhiThet{A, h) ; 

9 zO = [2; 1] ; 

10 z = zeros(2, N+1) ; u = zeros(2, N+1) ; 

11 z {:, 1) = zO ; 

12 u(:, 1) = P\z{:, 1) ; 

13 B = eye (2)-tht*phi*J ; C = eye {2)+(1-tht)*phi*J ; 

14 for j = 1 : N 

15 u(:, j + 1) = B\{C*u(:, j)) ; 

16 z(:, j + 1) = P*u(:, j + 1) ; 

17 end 

18 

19 uex = zeros (size (u)) ; %Exact transformed solution 

20 zex = zeros(size (z)) ; %Exact solution 

21 uex (:, 1) = u(:, 1) ; zex (:, 1) = z(:, 1) ; 

22 iP = inv (P) ; 

23 for j = 1 : N 

24 uex (:, j + 1) = expm (J*t(j + 1))*uex(:, 1) ; 

25 zex ( :, j+1) = P*uex(:, j + 1) ; 

26 end 

27 Errors = abs(z-zex) ' ; 

28 fprintf ( ' - \n') ; 

29 fprintf ( ' t\t Ix(t ) -xexact (t) I\t |y(t ) -yexact ( t )| \n ' ) ; 

30 fprintf ( '- \n’) ; 

31 for j = 1 : length(t) 

32 fprintf ( ' %2i\t %8.7e\t\t %8.7e\n', t(j), Errors{j, ... 

1), Errors{j, 2) ) ; 

33 end 

34 fprintf ( ' - \n') ; 

35 

36 plot{t, z(l, :), ' -b ' , t, z{2, :), '— m', 'LineWidth', 2) 

37 xlabel ( ' t' ) ; 

38 set (gea, 'XTick', linspace(a, b, 11)) ; 
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39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 


set (gea, 'YTick', linspace(0, 1, 11)) ; 
grid on ; 

legend ( ' x (t) ' , ' y (t) ' ) ; 

set (gea, 'fontweight' , 'bold') ; 

function [phi, tht, P, J] = CompPhiThet(A, h) 

[P, J] = jordan(A) ; 

L1 = J(l, 1) ; L2 = J(2, 2) ; 

if isreal(Ll)&& isreal(L2) 

if abs(Ll) > eps && abs(L2) > eps 
if abs(Ll-L2) > eps 

phi = (L1-L2)* (exp (Ll*h)-1)* (exp (L2*h)-1)/... 
(L1*L2* (exp (Ll*h) -exp (L2*h))) ; 

tht = (L2* (exp (Ll*h)-1)-Ll* (exp (L2*h)-1))/... 
((L1-L2)* (exp (Ll*h)-1)* (exp (L2*h)-1)) ; 

else 

phi = (exp (Ll*h)-1)"2/(L1" 2*h*exp (Ll*h)) ; 

tht = ( 1-exp (Ll*h)+L1 *h*exp (L1*h)).,. 

/ (exp (Ll*h)-1)"2 ; 

end 

end 

if abs(Ll) > eps && abs(L2) < eps 
phi = h ; 

tht = (exp (Ll*h)-1-Ll*h)/(Ll*h* (exp (Ll*h)-1)) ; 

end 

if abs(Ll) < eps && abs(L2) > eps 
phi = h ; 

tht = (exp (L2*h)-1-L2*h)/(L2*h* (exp (L2*h)-1)) ; 

end 

if abs(Ll) < eps && abs(L2) < eps 
phi = h ; 
tht = 0.5 ; 

end 

end 

end 


By exeeuting the script, the approximation errors at tj,j = 0,...,N are as 
follows: 


t 

|x(t)-xexact(t) 1 

|y(t)-yexact(t) 1 

0 

O.OOOOOOOe+00 

O.OOOOOOOe+00 

1 

5.5511151e-17 

1.1102230e-16 

2 

5.5511151e-17 

5.5511151e-17 

3 

8.3266727e-17 

1.1102230e-16 

4 

5.5511151e-17 

1.1102230e-16 

5 

5.5511151e-17 

8.3266727e-17 

6 

4.1633363e-17 

5.5511151e-17 

7 

2.7755576e-17 

4.1633363e-17 

8 

4.1633363e-17 

5.5511151e-17 

9 

3.4694470e-17 

4.8572257e-17 

10 

3.4694470e-17 

4.8572257e-17 
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The Python code of the script SolveLinSysExact. py is: 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 


import numpy as np 
from sympy import Matrix 
from scipy.linalg import expm 
import matplotlib.pyplot as plt 

def CompPhiThet{A, h): 

eps = np.spacing(1,0) 

P, J = Matrix(A).jordan.form() 

P, J = np.array(P, float), np.array(J, float) 

Ll, L2 = J[0, 0], J[l, 1] 

if isinstance (L1, float) and isinstance (L2, float) : 
if np.abs(Ll) > eps and np.abs(L2) > eps: 
if np. abs (L1-L2) > eps: 

phi = (L1-L2)* {np.exp(Ll*h)-1)*(np.exp(L2*h)-1)\ 
/(L1*L2 *(np.exp(Ll*h)-np.exp{L2 *h) ) ) 
tht = (L2*(np.exp(Ll*h)-1)-Ll*(np,exp{L2*h)-1))\ 
/{{L1-L2)*(np.exp(Ll*h)-1)*(np.exp(L2*h)-1)) 
else : 

phi = (np.exp{Ll*h)-1) "2/{Ll"2 *h*np,exp{Ll*h)) 
tht = (1-np.exp{Ll*h)+Ll*h*np.exp(Ll*h))\ 

/ (np.exp(Ll*h)-1)**2 

if np.abs(Ll) > eps and np.abs(L2) < eps: 
phi = h 

tht = (np.exp{Ll*h)-1-Ll*h)/{Ll*h*(np.exp{Ll*h)-1)) 
if np.abs(Ll) < eps and np.abs(L2) > eps: 
phi = h 

tht = (np.exp{L2*h)-1-L2*h)/{L2*h*(np.exp{L2*h)-1)) 
if np.abs(Ll) < eps and np.abs(L2) < eps: 
phi = h 
tht = 0.5 

return phi, tht, P, J 
a, b = 0.0, 10.0 

A = np.array{[[-3, 2], [1, -1]]) 

N = 10 

t = np.linspace(a, b, N+1) 
h = (b-a)/N 

phi, tht, P, J = CompPhiThet(A, h) 
zO = np.array([l, 0.5]) 

z, u = np.zeros((2, N+1), float), np.zeros((2, N+1), float) 
z [:, 0] = zO 

u[:, 0] = np.linalg.solve(P, z[:, 0]) 

B = np.eye(2)-tht*phi*J ; C = np.eye(2)+{1-tht)*phi*J 
for j in range (N): 

u[:, j+1] = np.linalg.solve(B,C@u[:, j]) 

z [ :, j + 1] = P@u[:, j + 1] 

uex, zex = np.zeros((2, N+1), float), np.zeros((2, N+1), float) 
zex[:, 0] = zO # Exact solution of the linear system 
uex[:, 0] =u[:, 0] 
for j in range (N): 

uex[:, j + 1] = expm(J*t[j + 1])@uex[: , 0] 
zex[:, j+1] = P@uex[:, j+1] 

Errors = np. abs (z-zex) 


54 
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55 print ( '- 

56 print ( ' t\t IX(t ) -xexact{t) I\t |y (t)-yexact(t) | ' ) ; 


) ; 


57 print ( '- 

58 for j in range ( len (t )): 

59 print (' {0 : 2 .Of} format {t[ j ]) , ' \t ' , 


) ; 


'{0: 8 .7e} format (Errors[0, j]), '\t\t' \ ... 

' {0; 8 .7e}' . format (Errors[l, j])) 


60 print ( ' 

61 


) ; 


62 plt. plot (t/ z [ 0, : ] , ' ~b ' , label= ' x (t) ' , Iw = 3) 

63 plt.plot(t, z[l, :], '— m', label= ' y (t) ' , lw= 3) 

64 plt.xlabel( 't' , fontweight= ' bold ’ ) ; 

65 plt.xticks(np.linspace(a, b, 11), fontweight= ' bold ' ) 

66 plt.yticks(np.linspace(0, 1, 11), fontweight= ' bold ' ) 

67 plt.grid(True, ls=':') 

68 plt. legend () 


8.3.2 Exact Difference Schemes for Nonlinear Equations 

In this section, exact finite difference schemes will be established to differential 
equations whose right-hand sides are of the form: 



( 8 . 8 ) 


where n > 2 is a positive integer. 

The exact solution of equation (8.8) is given by 



Substituting the initial condition, gives 


l — {n—l)atoUQ ^ 


Hence the exact solution is given by: 



(8.9) 


To derive the exact finite difference scheme of Equation (8.8), the substi¬ 
tutione: 


toand u->-Uk+\ 


are used, with the notice that tk+i —tk = h. Then, 
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Raising the two sides to power n—1 and dividing the numerator and denom¬ 
inator of the RHS by n — 1 and doing few manipulations give the form: 

uU-nl-^ = -a{n-l)hul-^ul-\ 


or 

/ \( n—2 I n —3 i i n—3 i n —2\ / i\l n—1 n — 1 

[Uk+i — Uk) + • • • + Uk-\~iUf, J — —a(n— 


from which the exact finite difference scheme 

n—1„ n—1 


^fc+1 '^k 

h 


-a{n-l)u^ 


u 


.n—2 
fc+1 ■ 


-U 


n—3 
fc+1 


Uk- 


n—3 I n—2 
-Uk+lUf, +Uk 


For n = 2, the exact finite difference scheme for 

du 9 , , 

* = <*> 


(890) 


IS 


^fc+1 ^k 




For n = 3, the exact finite difference scheme for 


du o. . 

a=<*> 


is 

Uk + l-Uk _ 

h Uk “t“ U/jj-i-i 

For n = 4, the exact finite difference scheme for 


du 

dt 


= —au^(f) 


is 

Uk+i — Uk _ —3QMfc^fc+i 

h ul+UkUk+l+ul_^_■^^ 

The Python script ExactFDMupown. py implements the exact finite differ¬ 
ence scheme for the differential equation (8.8), with uq = 0.5 and shows the 
errors at the discrete points tj,j = 0,..., 10: 


1 # ExactFDMupown . py 

2 import numpy as np 

3 

4 def ExactSol{t, n, uO): 

5 U = (n-1) *u0** {n-D / ( {n-D * {1+(n-1) *t*u0** (n-1) ) ) 

6 U = np.array ( [v**(1/(n-1)) for v in U]) 

7 return u 
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8 

9 def ExactFDM{t, n, h, uO): 

10 u = np. zeros-like (t) 

11 u [0] = uO 

12 for k in range (len (u)-1): 

13 u [k + 1 ] = . . . 

( (n-1) *u[k] ** (n-1) / { {n-D * (1+ (n-1) *h*u [k] ** (n-1) ) ) ) 
**(1/ (n-1)) 

14 return u 

15 

16 a, b, N = 0,0, 10.0, 5 

17 t = np.linspace(a, b, N+1) 

18 h, uO = (b-a)/N, 0.5 

19 ns = [2, 4, 8, 10] 

20 for n in ns: 

21 uex = ExactSol(t, n, uO) 

22 ufdm = ExactFDM(t, n, h, uO) 

23 Errors = np.zeros((2, N+1, float) 

24 Errors [ 0, : ] = t 

25 Errors[1, :] = np. abs (uex-ufdm) 

26 Errors = Errors.T 

27 

28 print (' Errors corresponding to n = '+str(n)) 

29 print ( ' - ' ) 

30 print ( ' h \t Error') 

31 print ( ' - ' ) 

32 for j in range (len (Errors)): 

33 print (' {0: 3 .2f} '. format (Errors[j][0]) + ' \t ' + ... 

' {0:1.8e}' . format (Errors[j][l])) 

34 print ( ' - \n\n’) 


Executing the code gives the following results: 

runfile(’D;/PyFiles/ExactFDMupown.py’, wdir=’D;/PyFiles’) 
Errors corresponding to n = 2 


t 

Error 

0.00 

O.OOOOOOOOe+00 

2.00 

O.OOOOOOOOe+00 

4.00 

O.OOOOOOOOe+00 

6.00 

O.OOOOOOOOe+00 

8.00 

O.OOOOOOOOe+00 

10.00 

1.38777878e-17 


Errors 

corresponding to n = 5 

t 

Error 


0.00 O.OOOOOOOOe+00 
2.00 O.OOOOOOOOe+00 
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4.00 O.OOOOOOOOe+00 
6.00 O.OOOOOOOOe+00 
8.00 O.OOOOOOOOe+00 
10.00 O.OOOOOOOOe+00 


Errors 

corresponding to n = 8 

t 

Error 

0.00 

O.OOOOOOOOe+00 

2.00 

O.OOOOOOOOe+00 

4.00 

5.55111512e-17 

6.00 

5.55111512e-17 

8.00 

5.55111512e-17 

10.00 

1.11022302e-16 


Errors 

corresponding to n = 10 

t 

Error 

0.00 

O.OOOOOOOOe+00 

2.00 

O.OOOOOOOOe+00 

4.00 

O.OOOOOOOOe+00 

6.00 

5.55111512e-17 

8.00 

5.55111512e-17 

10.00 

5.55111512e-17 


The code of the MATLAB script ExactFDMupown.m is: 


1 a = 0.0 ; b = 10.0 ; 

2 N = 5 ; 

3 t = linspace(a, b, N+1) ; 

4 h = (b-a)/N; 

5 uO = 0.5 ; 

6 ns = [2, 4, 8, 10] ; 

7 for n = ns 

8 uex = ExactSol(t, n, uO) ; 

9 ufdm = ExactFDM(t, n, h, uO) ; 

10 Errors = zeros(2, N+1) ; 

11 Errors (1, :) = t ; 

12 Errors (2, :) = abs (uex-ufdm) ; 

13 Errors = Errors’ ; 

14 fprintf (' Errors corresponding to n = %i\n', n) ; 

15 fprintf ( '- \n ' ) ; 

fprintf ( ' t \t Error\n') ; 


16 
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17 fprintf ( '-\n ' ) ; 

18 for j = 1 : N+1 

19 fprintf (' %2i\t%8 . 7e\n ' , Errors(j, 1), Errors(j, 2)) ; 

20 end 

21 fprintf ( ' - \n\n') ; 

22 end 

23 

24 function u = ExactSol(t, n, uO) 

25 U = (n-1) *u0" {n-D . / ( {n-D * {1+(n-1) *t*uO" (n-1) ) ) ; 

26 u = U. ^ (1/ {n-D ) ; 

27 end 

28 

29 function u = ExactFDM(t, n, h, uO) 

30 u = zeros (size (t) ) ; 

31 u (1) = uO ; 

32 for k = 1 : length(t)-l 

33 u (k+1) = . . . 

( (n-1) *u(k) ^ (n-1) / { {n-D * (1+ (n-1) *h*u (k) ^ (n-1) ) ) ) 
DI/(n-D) ; 

34 end 

35 end 


8.3.3 Exact Finite Difference Schemes for Differential Equa- 
tions with Linear and Power Terms 

Given a differential equation: 

du 

— = au{t) — = UQ, n>2 (8-11) 

dt 

To construet the exact finite difference scheme, the denominator function (f is 
derived from the linear term auft) as 


4>{h,a) = 


1 — e 


— ah 


a 


and the nonlocal approximation is used for the nonlinear term —j3vF(f) as 


n—2 I n—3 


^ fc +1 


n —3 

'Uk+lU^ 


,n-2 


Then, the exact finite difference scheme for the differential equation (8.11) 
will be: 


^fc+1 '^k 

l — e~ 


n—l^n—1 


-I3{n-l)ul Di 


n-2 , n-3 , , n-3 , n-2 

Uk+1 + «fe -I-1" + '^k 


A special case is when n = 2 which yields the logistic equation 

du 


( 8 . 12 ) 


dt 


= au(t) — j3u^{t), 
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^k+\ n 

_ ^^k P^k^k-\-l 


from which: 

_ {l + (j)a)uk 
l + 4>Puk 

The Python script ExactFDMLogistic. py solves the logistic model, using 
an exact finite difference scheme: 


1 

import numpy as np 



2 

import matplotlib.pYplot as 

plt 


3 

a, b, N = 0.0, 50.0, 100 



4 

t = np.linspace(a, b, N+1) 



5 

h = (b-a) /N 



6 

uO = 0.1 



8 

U = np.zeros-like(t) 



9 

u[0] = uO 



10 

alpha, beta = 0.25, 0.25 ; 



11 

phi = (1-np.exp(-alpha*h))/alpha ; 


12 

for k in range (N): 



13 

u[k+l] = (l+phi*alpha)*u 

[k]/(l+phi*beta*u[k]) ; 

14 

MaxError = np.linalg.norm(u- 

uex, np 

inf) 

15 

print (' Maximum error = MaxError) 


16 

plt.plot(t, u, color= ’ brown ' 

, lw=3) 


17 

plt.xlabel( ' Time (t)', fontweight = 

'bold' ) 

18 

plt.ylabel( ' Population u(t)' 

, fontweight = 'bold') 

19 

plt.grid(True, ls=':') 



20 

s = (b-a) /10 . 



21 

plt.xticks(np.arange(a, b+s. 

s), fontweight = 'bold') 

22 

plt.yticks(np.arange(0., 1.1 

, 0.1), 

fontweight = 'bold') 


By executing the code, the maximum error is shown and the solution of the 
model is explained in Figure 8.7. 


runfile(’D:/PyFiles/ExactFDMLogistic.py’, wdir=’D:/PyFiles’) 
Maximum error = 3.3306690738754696e-16 

The MATLAB script ExactFDMLogistic .m applies an exact finite differ¬ 
ence scheme to solve the logistic model: 


1 %ExactFDMLogistic . m 

2 a=0.0;b=50.0;N=100; 

3 t = linspace{a, b, N+1) ; h = (b-a)/N ; 

4 uO = 0.1 ; 

5 u = zeros(l/ N+1) ; 

6 alpha = 0.25 ; beta = 0.25 ; 

7 phi = (exp (alpha*h)-1)/alpha ; 

8 u {1) = u 0 ; 

9 uex = zeros(l, N+1) ; uex(l) = uO ; 
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FIGURE 8.7: Solution of the logistic model, using exact finite difference 
scheme. 


10 for k = 1 : N 

11 u(k + l) = {l+phi*alpha)*u (k)/( l+phi+beta*u (k)) ; 

12 uex (k+1) = . . . 

alpha*uO/( (alpha-beta*uO ) *exp (-alpha*t(k+1)) +beta*uO) ; 

13 end 

14 

15 MaxError = norm(u-uex, inf) ; 

16 disp([’Max Error = ’ num2str (MaxError)]) ; 

17 

18 plot(t, u, 'r', 'LineWidth', 3) ; 

19 xlabel ( ' Time (t)') ; 

20 ylabel ( ' Population u(t) ') ; 

21 set (gea, 'f ontweight ’ , 'bold') ; 

22 grid on ; 

23 s = (b-a) /10 ; 

24 set (gea, 'XTiek', a:s:b) ; 

25 set (gea, 'YTick', linspace(0, 1.1, 12)) ; 


8.4 Other Nonstandard Finite Difference Schemes 

In the case that an exact finite difference scheme cannot be obtained, then the 
purpose in constructing a nonstandard finite difference scheme is to establish 
the best possible finite difference scheme that solves the problem. 
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In [37] Mickens put the rules for selecting the denominator function and 
dealing with the nonlinear terms. Given an initial value problem, 

^ = f(u), u(0) = uo 

To find the denominator function, the equilibria points of the differential equa- 
tions are calculated by solving:/(M) = 0 for u(t). Let ui,U 2 , ■■■,Un be the equi¬ 
libria points of the differential equation. Let 


Tk 


'df 

du 


,k = 


U=Uf^ 




and r = max{|ri;|,/c = 1,... ,n}. Then the denominator function is given by: 


4'{r,h) 


r 


The linear and nonlinear terms at the right-hand side can be approximated 
using nonlocal approximations. For example a term u(t) in the differential 
equation can be approximated at tk by 2uk — Uk+i, a term u^{t) in the differ- 
ential equation can be approximated at tk by 2u\ — UkUk+i, etc. 


Example 8.4 This example is taken from [37] and constructs a nonstandard 
finite difference scheme for the differential equation: 

= u^it)-u^{t),u{0) = uo 

The equilibria points of the model are ui = U 2 = 0 and U 3 = 1. The denom¬ 
inator function is (j){h) = l — e~^. The term u‘^{t) is approximated at t^ by 
2u\ — UkUk+i and the nonlinear term —u^{t) is approximated by —u^Uk+i- 
The nonstandard finite difference scheme is: 


^fc+1 

<t> 




which after few manipulations gives: 

_ {l + 2(j)Uk)uk 
l + (j){uk+ul) 

The Python script MickPopModel. py implements the nonstandard finite dif¬ 
ference scheme of Mickens and plots the solution: 


1 import numpy as np 

2 import matplotlib.pYplot as plt 

3 a, b, N = 0,0, 25.0, 100 
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4 t = np.linspace(a, b, N+1) 

5 h = (b-a)/N 

6 uO = 0.1 

7 

8 u = np.zeros.like(t) 

9 u[0] = uO 

10 phi = l-np.exp(-h) 

11 for k in range (N) : 

12 u [k + 1] = { l + 2*phi*u [k]) *u[k] /( 1+phi*{u[k]+u[k] * *2 ) ) 

13 plt.plot(t/ u, color= ’ orangered ' , lw=3) 

14 plt.xlabel( 't' , fontweight = 'bold') 

15 plt.ylabel( ' u(t )' , fontweight = 'bold') 

16 plt.grid(True, ls=':') 

17 s = (b-a) /10. 

18 plt.xticks(np.arange(a, b+s, s), fontweight = 'bold') 

19 plt.yticks(np.arange(0., 1.1^ 0.1), fontweight = 'bold') 


By executing the code Figure 8.8 is obtained. 
The MATLAB code is 


1 a = 0.0 ; b = 25.0 ; N = 100 ; 

2 t = linspace(a, b, N+1) ; h = (b-a) /N ; 

3 uO = 0.1 ; 

4 u = zeros(l, N+1) ; 

5 phi = l-exp(-h) ; 

6 u (1) = u 0 ; 



0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 22.5 25.0 


FIGURE 8.8: Solution of the model u'{t) = v?'{f) — using Micken’s non¬ 
standard finite difference scheme. 
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7 for k = 1 : N 

8 u(k+l) = {l+2*phi*u (k) )*u(k)/(1+phi*{u(k)+u(k)"2)) ; 

9 end 
10 

11 plot(t, u, 'r', 'LineWidth', 3) ; 

12 xlabel ( ’ Time (t)') ; 

13 ylabel ( ' Population u(t) ') ; 

14 set (gea, 'f ontweight ’ , 'bold') ; 

15 grid on ; 

16 s = (b-a) /10 ; 

17 set (gea, 'XTiek', a:s:b) ; 

18 set (gea, 'YTick', linspaee(0, 1.1, 12)) ; 
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Solving Optimization Problems: Linear and 
Quadratic Programming 


Abstract 

Linear programming is one of the most important methods of mathematical 
programming and most applied in practical life to ensure optimal use of limited 
resources. Such examples include the optimal mix of the products produced 
by a particular piant to achieve the maximum profit according to the available 
labors and raw materials. Also, moving certain products from production areas 
to consumption centers so that each consumer center satisfies its demand at 
the lowest possible cost. 

This chapter is organized as follows. The first section discusses the form 
of a linear programming problem. The second and third sections discuss 
the Solutions of linear programming problems using the built-in functions in 
MATLAB® and Python. Section 4 uses the Python pulp package for solving 
linear programming problems. The package Pyomo is another Python package 
for solving linear programming problems, and is discussed in Section 5. Section 
6 uses the gekko Python package for solving linear programming problems. 
Section 7 is devoted to solving quadratic programming problems using many 
MATLAB and Python packages. 


9.1 Form of a Linear Programming Problem 

The general form of the linear programming problem is: 

min aiXi + a 2 X 2 -\ - \-anXn (9.1) 
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subject to equality constraints: 

aiiXi + ai2X2 +■■■ + ainXn = bi 
a2lXi + a22X2 +■■■ + a2nXn = &2 

031X1 +032^2+ ... + a3nXn = 63 (9-2) 


(^m\X\~\~ am2X2p •••P^mnXn — bm 
inequality constraints 


C 11 X 1 + C 12 X 2 + . 

■ ■ 

= di 


C2lXi + C22a:2 + - 

■ ■ ^2n^n 

= d2 


C3ia;i + C32X2 + . 

■ ■ H“ 

= ds 

(9.3) 


CllX\-\-€12X2 +■ ■-P CinXn — di 

and box constraints: 

XI G (xr",xp*),...,x„ G (9.4) 

It is worthy to notice that an inequality constraint of the form: 

n 

J=1 

is equivalent to the inequality constraint: 

n 

< -7j, 

j=i 

therefore, it is convenient to always write any inequality constraint in the form 

LHS < RHS. 

The linear programming problem described by equations (9.1)-(9.4) can 
be written in the matrix form as: 


T 

min a x 

(9.5) 

Ax = b 

(9.6) 

Cx < d 

(9.7) 

X e X 

(9.8) 


subject to constraints 



Farm of a Linear Programming Problem 
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FIGURE 9.1: Graphs of the functions f{x) = e and —f{x), 

which show that the value of x that maximizes f{x) minimizes —f{x). 


where, 


A = 

Oii 012 

021 022 

^2n 

,x = 

Xl 

X2 

,b = 

■ h ■ 
b2 


Oml ^ms 

^mn 


Xn 




c = 

Cll 

C21 

Cl2 

C22 

C\n 

C2n 

,d = 

1 

t-H (N 

^ ■ ■ 

1 _ 

,x = 



Ql 

^Is 

■ ■ ■ ^In 


. di _ 


( vmin vmax\ 

. \^n / _ 


The problem of maximizing some objective function f{x) is equivalent to 
the problem of minimizing —f(x), as the maximization problem of f{x) and 
the minimization problem of —f(x) have the same optimal solution x°^^. In 
Figure 9.1 the graphs of the functions f{x) = sin (^) and —f{x) are 

plotted. From the graphs of the two functions, it is ciear that the maximum 
value of f{x) is obtained at the same point that minimizes —f{x). 

Then, the problem: 

T 

max cy. x 

subject to constraints 

Ax = b 
Cx < d 
X G X 

is equivalent to the problem: 

min Id^^^x 
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subject to constraints 

Ax = b 
Cx < d 
X € X 

(/3 = —a) without further change in the equality or inequality constraints. 


9.2 Solving Linear Programming Problems with 
linprog 

Considering the problem: 

T 

min OL X, 

:cgR" 

subject to constraints 

Ax = b, 

Cx < d, 

Ib < x < ub, 

the MATLAB function linprog can be used for solving linear programming 
problems. It is of the form: 

[xopt, fval] = linprog(objfun, C, d, A, b, Ib, ub) 

where xO is an initial starting point. If the problem does not contain lower 
and upper bounds, then the form is: 

[xopt, fval] = linprog(objfun, C, d. A, b) 

If the problem does not contain equality constraints, then A and b are replaced 
by empty squared brackets []. The form will be: 

[xopt, fval] = linprog(objfun, C, d, [], [], Ib, ub) 

To show how to use the function linprog, the following example is con- 
sidered: 

Example 9.1 In this example, MATLAB’s function linprog will be used to 
solve the linear programming problem: 

max 3a;i + X 2 + 2 a ;3 


3X1 +X2 

< 

40 

Xl 

+ 2x3 

< 

60 


X2 + 2x3 

< 

60 

IV 

o 

X2 > 0,X3 

> 

0 


subject to: 
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The following MATLAB instructione are used to find the optimal solution: 

>> objfun = [-3; -1; -2] ; 

» C = [3, 1, 0; 1, 0, 2; 0, 1, 2] ; 

» d = [40; 60; 60] ; 

» Ib = [0; 0; 0] ; 

>> [xopt, fval] = linprogCobjfun, C, d, [], [], Ib, []) 

Optimal solution found. 


xopt = 

10 

10 

25 


fval = 


-90 

The MATLAB linprog function uses by default the dual-simplex algorithm. 
The choice of the algorithm can be changed through the optimoptions struet. 
For example to switch to the interior-point solver, and solve Example 9.1 
the following instructione can be used [17]. 

>> Options = optimoptions('linprog’, 'Algorith’, 'interior-point’); 
>> [xopt, fval] = linprogCobjfun, C, d, [], [], Ib, [], Options) 

Minimum found that satisfies the constraints. 

Optimization completed because the objective function is non- 
decreasing in feasible directions, to within the selected value 
of the function tolerance, and constraints are satisfied to 
within the selected value of the constraint tolerance. 

xopt = 

10.0000 

10.0000 

25.0000 

fval = 

-90.0000 

>> Options = optimoptions('linprog', 'Algorith', 

’interior-point-legacy’, 'Disp', 'Iter') ; 

>> [xopt, fval] = linprogCobjfun, C, d, [], [], Ib, [], Options) 
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Optimization terminated. 

xopt = 

10.0000 

10.0000 

25.0000 


fval = 

-90.0000 


Python has also a function linprog located in the scipy. optimize library. 
To use the linprog function, it shall imported from the optimize library of 
scipy [18]. This can be done as follows: 

In [1]; from scipy.optimize import linprog 

The Python function linprog form is close to the MATLAB’s linporg form. 
Its syntax is: 

OptSol = linprog(objfun, A_ub = C, b_ub = d, A_eq = A, b_eq = b, 
bounds = bnds,\ method=’optmethod’, options = optoptions) 

To solve Example 9.1 with Python, the following Python instructions can 
be used: 


In [1]: import numpy as np 

In [2]: import scipy.optimize as opt 

In [3]: objfun = np.array([-3, -1, -2]) 

In [4]: C = np.array([[3, 1, 0], [1, 0, 2], [0, 1, 2]]) 

In [5]: d = np.array{[40, 60, 60]) 

In [6]: bnds = [(0., None), (0., None), (0., None)] 

In [7]: OptSol = opt.linprog(objfun, A_ub = C, b_ub = d, bounds = bnds, \ 
options = ({"disp" :True]-)) 


Primal Feasibility 

1.0 

0.1157362093423 

0.01711151690354 

0.0001832497415752 

9.462525147506e-09 


Dual Feasibility 

1.0 

0.1157362093423 

0.0171115169036 

0.0001832497416004 

9.462525023246e-09 


Duality Gap 

1.0 

0.1157362093423 

0.0171115169036 

0.0001832497416007 

9.462524985793e-09 


Step 

0.8915842403063 

0.8667218148033 

0.9929871273485 

0.9999484444422 


Path Parameter 

1.0 

0.1157362093423 

0.01711151690367 

0.0001832497416554 

9.462524789912e-09 


Objective 

- 6.0 

-31.98924052603 

-76.59885574796 

-89.85504503568 

-89.99999255556 


Optimization terminated successfully. 
Current function value: -89.999993 


Iterations: 4 


In [8]: print(OptSol) 

con: array([], dtype=float64) 
fun: -89.99999255555932 

message: 'Optimization terminated successfully.’ 
nit: 4 

slack: array([2.26564158e-06, 5.24875590e-06, 7.23457025e-06]) 
status: 0 
success: True 

x: array([ 9.99999993, 9.99999794, 24.99999741]) 


The default solver of the linprog function is the interior-point method. 
But there are other two options of the method, which are the simplex and 
revised simplex methods. They are used as follows: 
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In [9]: OptSol = opt.linprog(objfun, A_ub = C, b_ub = d, bounds = bnds,\ 
method=’simplex’, options = ({"disp":True})) 

Optimization terminated successfully. 

Current function value: -90.000000 
Iterations: 3 


In [10]: print(OptSol) 

con: array([], dtype=float64) 
fun: -90.0 

message: 'Optimization terminated successfully.’ 
nit: 3 

slack: array([0., 0., 0.]) 
status: 0 
success: True 
x: array([10., 10., 25.]) 


In [11]: OptSol = opt.linprogCobjfun, A_ub = C, b_ub = d, bounds = bnds,\ 
method=’revised simplex’, options = ({"disp" :True}-)) 


Phase Iteration Minimum Slack 

1 0 40.0 

Phase Iteration Minimum Slack 

2 0 40.0 

2 1 0.0 

2 2 0.0 

2 3 0.0 

Optimization terminated successfully. 
Current function value: -90.000000 
Iterations: 3 


Constraint Residual Objective 
0.0 0.0 
Constraint Residual Objective 


0.0 

0.0 

0.0 

0.0 


0.0 

-40.0 

-86.66666666667 

-90.0 


In [12]: print(OptSol) 

con: array([], dtype=float64) 
fun: -90.0 

message: 'Optimization terminated successfully.’ 
nit: 3 

slack: array([0., 0., 0.]) 
status: 0 
success: True 
x: array([10., 10., 25.]) 


9.3 Solving Linear Programming Problems with 
fmincon MATLAB’s Functions 

The MATLAB function fmincon can be used to solve a linear programming 
problem. To solve Example 9.1 with the fmincon function, a function handle to 
the objective function (or a user-defined objective function) shall be initiated. 
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Also, an initial guess xO, the linear inequality and equality constraints and 
the variables bounds are passed to the fmincon function. 

The following MATLAB instructions can be used to solve Example 9.1 
using the fmincon function: 

» f = @(x) objfun’*x ; 

>> xO = [0.; 0.; 0.] ; 

» C = [3, 1, 0; 1, 0, 2; 0, 1, 2] ; 

» d = [40; 60; 60] ; 

» Ib = [0; 0; 0] ; 

>> [xopt, fval] = fminconCf, [0.;0; 0.], C, d, [] , [] , Ib, []) 

Local minimum found that satisfies the constraints. 

Optimization completed because the objective function is non- 
decreasing in feasible directions, to within the default value 
of the optimality tolerance, and constraints are satisfied to 
within the default value of the constraint tolerance. 

<stopping criteria details> 

xopt = 

10.0000 

10.0000 

25.0000 


fval = 
-90.0000 


9.4 Solving Linear Programming Problems with pulp 
Python 

Python possesses a library pulp to solve linear programming problems [41]. 
Installing pulp is easy and can be done by typing: 

>>> pip install pulp 

In anaconda it can be installed by typing: 

In [13]: conda install -c conda-forge pulp 

To use pulp, the pulp library must be imported first: 


In [14]: import pulp as plp 
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Then, some variable (for example LPP) shall be used to define the problem and 
solve it. pulp enables the user to name his/her problem and defining its type 
{maximization, minimization) through the function LpProblem. To define a 
maximization problem the parameter LpMaximize shall be passed as a second 
argument to LpProblem, and to define a minimization problem the parameter 
LpMinimize shall be passed. For example: 

In [15]: LPP = plp.LpProblem("Problem of maximizing profit", 
LpMaximize) 

The variables of the problem shall be defined by using the pulp function 
LpVariable. It receives a string and bounds of the variable. For example if 
the problem contains a variable 0 < a; < 5: 

X = plp.LpVariable("x" , 0., 5.) 

Those variables can be used with problem instance LPP. 

Next, the equality and inequality constraints of the problem are added to 
the problem instance. If the problem contains an inequality constraint 2x + y < 
10 then this constraint is added to the problem using an instruction: 

LPP += 2*x + y <= 10 

In pulp the user does not have to change the forms of the inequality con¬ 
straints. if the problem contains a constraint x + y>2, it can be added to the 
problem instance directly without transforming it to other form: 

LPP += x + y >= 2 

Finally, after defining the whole problem, the solve () method of the LPP 
instance can be used to solve the problem and displaying the results. 

The Python script SolveExLinProg.py is used to solve the problem of 
Example 9.1 and show the results: 


1 # SolveExLinProg.py 

2 import pulp as plp 

3 LPP = plp.LpProblem (' Problem statement: \n', plp.LpMaximize) 

4 X, Y, z = plp.LpVariable( "x" , lowBound=0.), ... 

plp.LpVariable( "y " , lowBound=0 .), plp.LpVariable {"z" , ... 

lowBound=0.) 

5 LPP += 3*x + y + 2*z 

6 LPP += 3*x + y < 40 

7 LPP += x + 2*z < 60 

8 LPP += y + 2*z < 60 

9 LPP += X > 0. 

10 LPP += y > 0. 

11 LPP += z > 0. 

12 print (LPP) 

13 status = LPP.solveO 

14 print (’ Status of the optimization process: ' + ... 

plp.LpStatus[status]) 
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15 print ( ' Optimal maximizer found at: \n', plp.value (x), '\n', ... 

plp.value(y), '\n', plp.value(z)) 

16 print (' Optimal value of the objective function: ... 

plp.value(LPP.objective) ) 

Executing the code gives the following results: 

runfile(’D;/PyFiles/SolveExLinProg.py’, wdir=’D;/PyFiles’) 
Problem statement; 

MAXIMIZE 

3*x + l*y + 2*z + 0 
SUBJECI TO 


Cl: 

3 

X + y 

<= 40 

C2; 

X 

+ 2 z 

<= 60 

C3; 

y 

+ 2 z 

<= 60 

C4; 

X 

>= 0 


C5; 

y 

>= 0 


C6; 

z 

>= 0 



VARIABLES 
X Continuous 
y Continuous 
z Continuous 

Status of the optimization process; Optimal 
Optimal maximizer found at; 

10.0 

10.0 

25.0 

Optimal value of the objective function: 90.0 


9.5 Solving Linear Programming Problems with pyomo 

Pyomo is a Software based on Python to model and solve optimization problems 
[23]. There are two kinds of modelling presented by pyomo: concrete models 
where data are specified first before model construction and abstract models 
where the model is constructed first before data specification and the model 
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data are passed to the model during the runtime of the optimization pro- 
cess. This section presents the creation of concrete models for solving linear 
programming problems. 

In anaconda, pyomo can be installed by using the command: 
conda install -c conda-forge pyomo 

It is important to be sure that the targeted optimization solver is installed 
before proceeding to the use of pyomo. For example, if the ’glpk’ solver is 
not installed, it can be installed by typing the command: 

conda install -c conda-forge glpk 

The first step for solving a linear programming problem is to prepare the 
pyomo environment and solver, through importing both. This can be done by: 

import pyomo as pym 

from pyomo.opt import SolverFactory 

Then a variable of either type ConcreteModel or AhstractModel is constructed 
by using: 

Model = pym.ConcreteModel0 
or 

Model = pym.AbstractModelO 

If for example 1 < a; < 5 and 0 < y < oo are two variables of the optimization 
problem, they can be declared by using: 

Model. X = pym.Var(bounds=(l., 5.)) 

Model.y = pym.Var(within=NonNegativeReals) 

The objective function is defined by using the pym.Objective method which 
receives the mathematical expression of the objective function (for example 
10 X + y/100): 

Model.objfun = pym.ObjectiveCexpr = 10.*Model.x+Model.y/100) 

If the problem contains a constraint lOx + y < 100, this constraint can be 
added to the model by typing: 

Model.coni = pym.Constraint(expr=10*Model.x + Model.y <= 100) 

After completely defining the model, an optimization solver is used to solve 
the problem defined by the model. 

To solve Example 9.1 using pyomo, a Python script SolveExOWithPyomo. py 
is used. Its code is: 


1 import pyomo.environ as pym 

2 from pyomo.opt import SolverFactory 

3 LPP_model = pym.ConcreteModel() 

4 LPP_model.x, LPP_model.y, LPP_model.z = pym.Var(bounds=(0., ... 

None)), pym,Var(bounds=(0., None)), pym.Var(bounds=(0., None)) 
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5 LPP_model.objfun = ... 

pym.Objective(expr=3*LPP_model.x+LPP_model.y+2*LPP_model.z, 
sense=maximize) 

6 LPP_model.coni = pym.Constraint(expr=3*LPP_model.x + ... 

LPP_model.y < 40) 

7 LPP_model.con2 = pym.Constraint(expr=LPP_model.x + ... 

2*LPP_model.z < 60) 

8 LPP_model.con3 = pym.Constraint(expr=LPP_model.y + ... 

2*LPP_model. z < 60) 

9 opt = SolverFactory( ’ glpk ' ) 

10 OptResults = opt.solve(LPP_model) 

11 print ( ' \nStatus of optimization process: ... 

OptResults.solver.Status, '\n') 

12 print (' Status of termination condition: ... 

OptResults.solver.termination.condition, ' \n ' ) 

13 print (’ Optimal solution obtained at: \n x\t', ... 

LPP_model.X.value, ' \n y = ', LPP_model.y.value, ' \n ... 

z\t', LPP_model.z.value, '\n') 

14 print ('Value of the objective function at optimal solution: ', 

LPP_model.obj fun()) 


By executing the code, the following results are obtained: 
Status of optimization process: ok 

Status of termination condition: optimal 


Optimal solution obtained at: 


X 

10.0 


y 

10.0 


z 

25.0 


Value 

of the objective function at optimal solution: 

90.0 


9.6 Solving Linear Programming Problems with gekko 

Optimization problems are solved in gekko at mode 3 [2]. To solve Example 
9.1 with gekko, the Python script SolveEx2LPWithGekko.py is used: 


1 from gekko import GEKKO 

2 m = GEKKO() 

3 X = m.Var(l., 0, None) 

4 y = m.Var(l., 0, None) 

5 z = m.Var(l., 0, None) 

6 m.Obj(-3*x-y-2*z) 

7 m.Equation(3*x + y < 40) 

8 m.Equation(x + 2*z < 60) 

9 m.Equation(y + 2*z < 60) 
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10 m.options.IMODE = 3 

11 xopt=m. solve () 

12 print (’ Solutiori found at \n') 

13 print ('{0:10.7f}' . format (x[0])) 

14 print ('{0:10.7f}' . format (y[0])) 

15 print ('{0:10.7f}' . format (z[0])) 

16 print ('Value of the objective function: 

' {0: 10 .7f}' . format {3*x[0]+y[0]+2*z[0]) ) 


Executing the code gives: 

EXIT; Optimal Solution Found. 

The solution was found. 

The final value of the objective function is -89.9999999911064 


Solver ; IPOPT (v3.12) 

Solution time ; 9.100000001126318E-003 sec 

Objective ; -89.9999999911064 

Successful solution 


Solution found at 

10.0000000 

10.0000000 

25.0000000 

Value of the objective function: 90.0000000 


9.7 Solving Quadratic Programming Problems 

A quadratic programming problem is an optimization problem whose objective 
function is quadratic and constraints are linear. A general form of a quadratic 
programming problem is: 

min -x'^Hx + ot^x, ff € a and a; S R" (9-9) 

together with the linear equality constraints described by (9.2) and linear 
inequality constraints described by Equation (9.3). 

Through this section, solution of quadratic programming problem 
described in the following example will be considered: 
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Example 9.2 

min x\ + 4^2 + + x\ + 4 xiX 3 + 4 x 2 X 4 + 2 xi + X 2 + X 3 + 2 x 4 

subject to the inequality constraints: 


xi + X4 

> 

10 

d + ^2-1-X3-1-2X4 

> 

24 

X 2 + X 3 

> 

20 

X 4 + X 2 + X 3 + X 4 

< 

30 

0 < Xi,X2,X3,X4 

< 

20 


MATLAB has a function quadprog for solving quadratic programming prob¬ 
lems. The function fmincon can also be used for solving quadratic problems. 


Solutiori with quadprog MATLAB’s function 

The MATLAB’s function quadprog receives the matrix H, the vector a, 
matrix of inequality constraints C and its right-hand side vector d, matrix of 
equality constraints A and its right-hand side b, and vectors of lower bounds 
Ib and upper bounds Ub. It is of the form: 

[xopt, fval] = quadprogCH, alpha, C, d, A, b, Ib, ub) ; 

If any of the inputs is missing, it can be replaced by blank squared brackets 
[]. The MATLAB script SolveQuadProg.m is used to solve Example 9.2. 


1 % SolveQuadProg.m 

2 ciear ; clc ; 


3 

H = 

[2, 0 

. 4, 

0 

0 

00 

0 

0 

00 

0 

0 

Jh> 

4 

alp 

= [2; 

1; [ 

L; 2] ; 

5 

C = 

[-1, 

-0, - 

-0, -1; -2, -1, -1, -2; 0, -1, -1 

6 

d = 

[-10; 

-24; 

; -20; 30] ; 

7 

Ib = 

[0; 

0; 0; 

: 0] ; 

8 

ub = 

[20; 

20; 

20; 20] ; 


9 [xopt, fval] = quadprog(H, alp, C, d, [], [], Ib, ub) ; 

10 fprintf ( ’ Optimal solution found at: \n') ; disp(xopt) ; 

11 fprintf (’ Objective function at optimal point : %10.7f\n', fval) ; 


Executing the codes gives: 

Minimum found that satisfies the constraints. 


Optimization completed because the objective function is non- 
decreasing in feasible directions, to within the default value 
of the optimality tolerance, and constraints are satisfied to 
within the default value of the constraint tolerance. 
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<stopping criteria details> 

Optimal solution found at: 
5.0000 
10.0000 
10.0000 
5.0000 


Objective function at optimal point: 1290.0000000 

Solution with the MATLAB’s function fmincon 

The MATLAB’s script SolveQPWithfmincon.m is used to solve the Example 

9 . 2 . 


1 

ciear ; clc ; 




2 

H = [2, 0, 4, 0; 0, 8, 0, 

4; 4, 0, 8, 0; 

0 

C\1 

O 

3 

alp = [2; 1; 1; 2] ; 




4 

objfun = @(x) 0.5*x'*H*x 

+ alp'*x ; 



5 

CNJ 

1 

I—1 

1 

o 

1 

o 

1 

1 

II 

U 

-1, - 1 , -2; 0, 

-1 

-1, 0; 1, 1, 1 , 1] ; 

6 

d = [-10; -24; -20; 30] ; 




7 

Ib = [0; 0; 0; 0] ; 




8 

ub = [20; 20; 20; 20] ; 




9 

xO = [1; 1; 1; 1 ] ; 




10 

[xopt, fval] = fmincon(objfun, xO, C, d. 

[ 

, [], Ib, ub) ; 

11 

fprintf ( 'Optimal solution 

found at : \n ' ) 


disp (xopt) ; 

12 

fprintf (’ Objective function at optimal point: %10.7f\n', fval) ; 


Executing the code gives: 


Local minimum found that satisfies the constraints. 

Optimization completed because the objective function is non- 
decreasing in feasible directions, to within the default value 
of the optimality tolerance, and constraints are satisfied to 
within the default value of the constraint tolerance. 

<stopping criteria details> 

Optimal solution found at: 

5.0000 

10.0000 

10.0000 

5.0000 


Objective function at optimal point: 1290.0000000 
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Solutiori with gekko Python 

The Python script SolveQPWithGekko. py solves the problem of Example 9.2: 


1 

from gekko import GEKKO 


2 

m = GEKKO() 


3 

xl = m.Var(1., 0, 20) 


4 

x2 = m.Var(l., 0, 20) 


5 

x3 = m.Var(l., 0, 20) 


6 

x4 = m.Var(1., 0, 20) 


8 

m.Obj(xl**2 + 4*x2**2+ 4*x3**2 + x4**2 + 
2*xl + x2 + x3 + 2*x4) 

4*xl*x3 + 4*x2*x4 + ... 

9 

m.Equation(xl+ x4 > 10) 


10 

m.Equation(2*xl + x2 + x3 + 2*x4 > 24) 


11 

m.Equation(x2 + x3 > 20) 


12 

13 

m.Equation(xl + x2 + x3 + x4 < 30) 


14 

m.options.IMODE = 3 


15 

xopt=m.solve() 


16 

print (’ Solution found at \n ' ) 


17 

print ( '{0: 10 . 7f }' . format (xl[0])) 


18 

print { '{0: 10 . 7f }' . format (x2[0])) 


19 

print ( '{0: 10 .7f}' . format (x3[0])) 


20 

print ( '{0: 10 .7f}' . format (x4[0])) 


21 

xopt = xl[0]**2 + 4*x2[0]**2+ 4*x3[0]**2 

+ x4[0]* * 2 + ... 


4*xl[0]*x3[0] + 4*x2[0]*x4[0] + 2*xl 
2*x4[0] 

[0] + x2 [0] + x3 [0] + . . . 

22 

print ('Value of the objective function: 
'{0:10. 7f}' , format (xopt)) 

, ... 


Results of execution: 

EXIT: Optimal Solution Found. 

The solution was found. 

The final value of the objective function is 1289.99999909229 


Solver : IPOPT (v3.12) 

Solution time : 1.630000000295695E-002 sec 

Objective : 1289.99999909229 

Successful solution 


Solution found at 

4.9999999 

10.0000000 
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10.0000000 

5.0000000 

Value of the objective function; 1289.9999991 

Solutiori with pyomo Python 

The GNU Linear Programming Kit (glpk) solver cannot be used for solving 
a quadratic programming problem in pyomo. Since pyomo does not include 
any solvers, it is important that the user installs a suitable solver that is 
compatible with pyomo. Such an example is the IPOPT solver. To install the 
IPOPT solver in anaconda, the following command can be used: 

conda install -c conda-forge ipopt 

The Python script SolveQPWithpyomo .py solves Example 9.2: 


1 import pyomo.environ as pym 

2 from pyomo.opt import SolverFactory 

3 m = pym.ConcreteModel() 

4 m.xl, m.x2, m.x3, m.x4 = pym.Var(bounds=(0., 20)), ... 

pym.Var(bounds={0., 20)), pym.Var(bounds=(0., 20)), ... 

pym.Var(bounds={0. , 20) ) 

5 m.objfun = pym.Objective(expr=m.xl**2 + 4*m.x2**2+ 4*m.x3**2 + ... 

m.x4**2 + 4*m.xl*m.x3 + 4*m.x2*m.x4 + 2*m.xl + m.x2 + ... 
m.x3 + 2*m.x4) 

6 m.conl = pym.Constraint(expr=m.xl+ m.x4 > 10) 

7 m.con2 = pym.Constraint(expr=2*m.xl + m.x2 + m.x3 + 2*m.x4 > 24) 

8 m.con3 = pym.Constraint(expr=m.x2 + m.x3 > 20) 

9 m.con4 = pym.Constraint(expr=m.xl + m.x2 + m.x3 + m.x4 < 30) 

10 opt = SolverFactory (' ipopt ' ) 

11 OptResults = opt.solve(m) 

12 print ( ' \nStatus of optimization process: ', ... 

OptResults.solver.Status, '\n') 

13 print (' Status of termination condition: ', ... 

OptResults.solver.termination_condition, ' \n ' ) 

14 print (’ Optimal solution obtained at: \n x\t', \ 

15 '{0: 8 .6f} format (m.xl,value), '\n Y = 'r ... 

'{0: 8 .6f} '. format {m.x2.value), '\n z\t', ... 

' {0: 8 . 6f }' . format {m.x3.value),\ 

16 '\n z\t', '{0: 8 .6f} '. format (m.x4.value) , '\n') 

17 print ('Value of the objective function at optimal solution: ', ... 

' {0: 8 . 6f }' . format (m.objfun() )) 


Executing the code gives the results: 

Status of optimization process: ok 

Status of termination condition: optimal 
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Optimal solution obtained at; 


X 


5.000000 

y 

= 

10.000000 

z 


10.000000 

z 


5.000000 


Value of the objective function at optimal solution: 1289.999975 



10 


Solving Optimization Problems: Nonlinear 
Programming 


Abstract 

Nonlinear programming problems are divided into two kinds of problems: 
unconstrained and constrained [19, 20]. An unconstrained optimization 
problem is a problem of the form 

min/(a;), x = [xi,X 2 ,-■ ■ ,XnP (10-1) 

A general constrained optimization problem consists of the objective func- 
tion in Equation (10.1) with equality constraints: 

E{x)^0, (10.2) 

and inequality constraints: 

J(a;)<0 (10.3) 

where x £ R", E{x) £ R^ and I{x) £ R'^. 

In this chapter, methods for solving both unconstrained and constrained 
optimization problems using MATLAB® and Python will be implemented. 


10.1 Solving Unconstrained Problems 

In this section, problem (10.1) is considered. If x* = [x\,X 2 , ■ ■ ■ ,xp^ £ R" is 
the optimal solution of the problem, then 

g{x*) = 0 (10.4) 

y^H{x*)y > 0, Vy£R", (10.5) 
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where g{x) = Wf{x) is the gradient vector and H{x) is the Hessian matrix 
defined by: 


■ 

dx\dxn 

aV(^) 

dx2dxn 

dxl J 

Equation (10.5) is to say that the Hessian matrix is a positive 
semi-definite. Equations (10.4)-(10.5) are necessary optimality conditions 
of X* . If condition (10.5) is replaced by the condition: 

y'^ff(a;*)y>0, VyeR" (10.6) 

(that is the Hessian matrix is positive definite), then the conditions 
(10.4)-(10.6) are sufficient optimality conditions. 

The numerical methods for solving (10.1) are iterative. They agree in start- 
ing from an initial guess x^^'> of the solution x* and then construet a sequence 

of Solutions x^^\x^^\x^‘^\ ..where f{x^^^) > /(a:^^)) >_At iteration k, 

moving from the point x^^') to a point in numerical optimization gen- 

erally passes through two steps: in the first step a search direction is 
determined, then in the second step a line search in the direction of 
(where || = 1) to locate the minimum point Pap^^'^ in 

the line is carried-out, such that /(x*^*“*“^^) < /(x^^^). The iterative process 
stops at a solution if 

||g(x('=))|| <£, 

where e > 0 is an arbitrary small positive real and H{x^^'l) is positive semi- 
definite [3, 20]. 

The numerical optimization techniques for unconstrained optimization 
care about two problems: 

(i) Line search problem: given a function /(x), its gradient g{x) = 
V/(x), a point x^^'> at iteration k a descent direction p^^\ find > 0 
such that for a = the function / is minimized along the ray 
+ap^^\ that is 

= argmin{/(x*^*^ + ap^^^)}. 
a>0 


g{x) = 


H{x) = 


V/(x) = 

r dfi^) 1 

dxi 

df(x) 

dx2 

r 

dfix) 

L dx„ -I 

d^f(x) 

dxl 

dx\dx2 

d^f(x) 

d^fix) 

dx2dxi 


d^f(x) 

d^fix) 

_ dxndxi 

dx„dx2 
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(ii) Search directiori problem: given a function f{x) and point at 
iteration k, find a unit vector such that +ap^^'^) is a decreas- 

ing function for 0 < a < amax- That is is a descent direction. 

The numerical optimization methods differ from each other by the methods 
through which the search directions are determined and the gradient vector 
and Hessian matrices are approximated and (or) updated. 

10.1.1 Line Search Algorithm 

There are two kinds of line search algorithms: exact line search and inexact 
line search algorithm. The exact line search algorithm looks for the exact step 
size a {a = aExact) such that 

aExact = argmin{/(a:('=) 
a>0 

which is numerically impractical. The inexact line search algorithms look 
for approximations to the exact step size aExact- This is done iteratively, 
starting from an initial guess and constructing a sequence of step sizes 
such that a* « aExact [49]. 

The most famous inexact line search algorithm is the hacktracking line 
search algorithm, which employs two parameters a and b and is based on 
Armijo condition: 

/(a:W < f{x) + aa^g{x^’^^fp^’^\ 

The parameter a is associated with the termination condition of Armijo. The 
parameter 0 < 6 < 1 is used to update the step size in each iteration, where 
a^ = ba^~^, starting from a large step size a*^(usually = 1). 

The MATLAB function LineSearch.m receives a function /, a vector g 
representing the gradient of / the starting vector x and a unit (direction) 
vector p and returns the optimal step size a: 


1 function alpha = LineSearch(f, g, x, p) 

2 a=0.3;b=0.9; 

3 alpha =1.0 ; 

4 while f (x+alpha*p) > f(x) + a*alpha*g(x) '*p 

5 alpha = b*alpha ; 

6 end 

7 end 


The Python code of the function LineSearch. py is: 


1 import numpy as np 

2 def LineSearch(f, q, x, p): 

3 a, b= 0.3, 0.9 

4 alpha = 1.0 

5 while f(x+alpha*p) > f(x) + a*alpha*np.dot(g(x), p): 
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6 alpha *= b 

7 return alpha 


10.1.2 The Steepest Descent Method 

From the elementary calculus, the gradient vector g{x) — V f{x) points to the 
direction of greatest increase in a function of several variables f{x). Hence, 
—g{x) points to the direction of greatest decrease in f{x). Any descent direc¬ 
tion p{x) makes an acute angle with —g{x). The steepest descent method 
chooses the search direction to be p{x) = — ||g[^]|| ■ Hence, in the steepest 

descent method the iterative process to progress from a point x^^'> at itera- 
tion fc to a point is given by the formula: 

||g(xW)|| 

where is found from a line search algorithm [20, 49]. 

For example, to minimize the function 


f{xi,X2) =xl-XiX2+4,X2 + l 

the gradient at any point x = [x\,X 2 \"^ is given by: 


9{x) 


2xi — X 2 
—x\ -|- 8 a :2 


The MATLAB script SolveExlwithSteepDesc.m finds the minimum of 
the given objective function based on the steepest descent method. 


1 % SolveExlwithSteepDesc.m 

2 ciear ; clc ; clf ; 

3 f = @(x) x(l)"2 -x(l)*x(2)+ 4*x{2)^2 + 1 ; 

4 g = @(x) [2*x{l)-x{2) ; -x {1)+8*x (2 ) ] ; 

5 xO = [1; 1] ; Eps = le-8 ; 

6 [x, Iterations] = GradDec(f, q, xO, Eps) ; 

7 disp (’ Optimum solution = ') ; disp(x) ; 

8 disp ([’ Iterations = ' num2str (Iterations)]) ; 

9 

10 function [x, Iterations] = GradDec(f, q, xO, Eps) 

11 X = X 0 ; 

12 Iterations = 0 ; 

13 while norm(g(x), 2) > Eps 

14 p = -g (x) /norm (g (x) , 2) ; 

15 alpha = LineSearch(f, q, x, p) ; 

16 X = X + alpha * p ; 

17 fprintf ( ’ %3i\t\t%14 . 9f \t\t%12 . 10e\n ' , Iterations, f (x), ... 

norm (g(x), 2)) ; 

18 Iterations = Iterations + 1 ; 

19 end 
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20 end 

21 

22 function alpha = LineSearch(f, g, x, p) 

23 a = 1 - 2 / (1 + sqrt (5) ) ; 

24 b = 2 / (l + sqrt (5) ) ; 

25 alpha = 1.0 ; 

26 while f(x+alpha*p) > f(x) + a*alpha*g(x)'*p 

27 alpha = b*alpha ; 

28 end 

29 end 


By executing the code, the optimal solution is obtained in 30 iterations. 


Iteration 

f(x) 

11g(x)11 

0 

1.728932188 

1.8761048104e+00 

1 

1.288589321 

1.1953530385e+00 

2 

1.116213769 

7.5167383791e-01 

29 

1.000000000 

3.1130274879e-08 

30 

1.000000000 

4.5572657815e-09 


Optimum solution = 
l.Oe-09 * 

-0.306552140415857 

0.513653144168223 

Iterations = 31 

The corresponding Python script SolveExlwithSteepDesc. py is: 


1 # SolveExlwithSteepDesc . py 

2 import numpy as np 

3 def LineSearch(f, g, x, p): 

4 a, b = 1-2/{1+np.sqrt{5)), 2/ (1+np,sqrt{5)) 

5 alpha = 1.0 

6 while f (x+alpha*p) > f(x) + a*alpha*np.dot(g(x), p): 

7 alpha *= b 

8 return alpha 

9 

10 def GradDec(f, g, xO, Eps) : 

11 X = xO ; 

12 Iterations = 0 ; 


13 print ( '-' ) 

14 print ( ' Iteration\t f (x)\t | |g(x) | | ' ) 

15 print ( ' - ’ ) 

16 while np.linalg.norm(g(x), 2) > Eps: 
p = -g(x)/np.linalg.norm(g(x), 2) 


17 
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18 alpha = LineSearch(f, g, x, p) 

19 X = X + alpha * p 

20 print (' {0: 5 .Of} format (Iterations), \ 

21 ' {0: 12 .lOf} format {f(x)),\ 

22 '{0:10.8e}'. format (np.linalg.norm(g(x)))) 

23 Iterations += 1 

24 print ( '-' ) 

25 return x, Iterations 

26 

27 f = lambda x: x[0]**2-x[0]*x[1]+4*x[1]+ 1 

28 g = lambda x: np. array ( [ 2*x [0]-x[1] , -x[0] +8*x [1]]) 

29 xO = np.array([l, 1]) 

30 Eps = le-8 

31 X, Iterations = GradDec(f, g, xO, Eps) 

32 print ( ' X = ' , x) 

33 print (' Iterations ^ Iterations) 

The disadvantage with the steepest descent method is that at the begin- 
ning it converges quickly to the optimum solution, but as it comes closer to 
the solution its convergance rate drops rapidly and its progress towards the 
optimum solution becomes too slow [3]. 

10.1.3 Newton’s Method 

At iteration k, the function / is approximated near by a quadratic formula 
from Taylor series as: 

f{x) « - *('=)) + (x- (10.7) 

The quadratic function at the right-hand side has a unique minimizer, 
obtained by differentiating the RHS of Equation (10.7) with respect to x 
and equating the resulting linear equation to 0: 

gix^’^'>) + H{x^^^){x - x^^">) = 0 

Setting X = and solving for gives: 

a.(fe+i) = -iT-yx('=))t/(x('=)). 

Hence, in Newton’s method the search direction at iteration k is given by: 

p(fc) ^__fj-l(3,(fe))g(3,(fe))^ 


and 

3,(fe+i) =a.(fe) + Q,(fe)p(fe) = a,(fe)_Q,(fe)_H--i(3,(fe))g(3.(fe)) 

Example 10.1 In this example, Newoton’s method will be used to find the 
optimum solution of the unconstrained minimization problem: 

min Sxj ++51oge 

a;^R2 2 
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The MATLAB script MinEx2WithPureNewton.m implements a function 
PureNewton to apply the Newton’s iterations for solving the minimization 
problem: 


1 % MiWithPureNewton .m 

2 ciear ; clc ; clf ; 

3 f = @ (x) 5 *x{l)"2 + x(2)''2/2 + 5*log (1 + exp {-X (1)-X (2) ) ) ; 

4 g = @(x) [10*x{l) - 5*exp{-x{l) - x(2) )/ {exp (-x (1) - x{2)) + 1); 

5 x{2) - 5*exp{-x{l) - X (2) ) / {exp {-X (1) - x(2)) + 1)] ; 

6 H = @(x) [5* (5*exp(x(l) + x(2)) + 2*exp(2*x(l) + 2*x(2)) + ... 

2) / (2*exp (x(l) + x(2)) + exp (2*x(l) + 2*x(2)) + 1) ... 

7 5* exp (x(l) + x(2)) / (2 *exp (x(l) + x(2)) + exp (2*x(l) + 2*x(2)) ... 

+ 1 ); 

8 5* exp (x(l) + x(2)) / (2 *exp (x(l) + x(2)) + exp (2*x(l) + 2*x(2)) ... 

+1) ... 

9 (7 *exp (x(l) + x(2)) + exp (2*x(l) + 2*x(2)) + 1) / (2 *exp (x (1) + . . . 

x(2)) + exp(2*x(l) + 2*x(2)) + 1)] ; 

10 xO = [10; 10] ; Eps = le-8 ; 

11 [x, Iterations] = PureNewton(f, q, H, xO, Eps) ; 

12 disp (’ Optimum solution = ') ; fprintf ( ' %15 . 14e\n%15 . 14e\n ' , ... 

x(l) , x(2) ) ; 

13 disp ([' Iterations = ’ num2str (Iterations)]) ; 

14 

15 function [x, Iterations] = PureNewton(f, q, H, xO, Eps) 

16 x = xO ; 

17 Iterations = 0 ; 

18 fprintf ( ' - \n’) ; 

19 fprintf (' Iteration\t\t f (x) \t\t\t ||g(x)||\n') ; 

20 fprintf ( '- \n ’ ) ; 

21 fprintf (' %5i\t\t%14 . 9f\t\t%12 . 10e\n ' , Iterations, f(xO), ... 

norm (g (xO), 2)) ; 

22 while norm(g(x), 2) > Eps 

23 p = -H(x)\g(x) ; 

24 alpha = LineSearch(f, g, x, p) ; 

25 X = X + alpha*p ; 

26 Iterations = Iterations + 1 ; 

27 fprintf (' %5i\t\t%14 . 9f\t\t%12 . 10e\n ' , Iterations, f(x), ... 

norm (g (x), 2)) ; 

28 end 

29 fprintf ( '- \n ' ) ; 

30 end 

31 

32 function alpha = LineSearch(f, g, x, p) 

33 a = 1-2/(1 + sqrt (5) ) ; 

34 b = 2/(1 + sqrt (5) ) ; 

35 alpha = 1.0 ; 

36 while f (x+alpha*p) > f(x) + a*alpha*g(x) '*p 

37 alpha = b*alpha ; 

38 end 

39 end 
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By running the code, the minimization problem is solved in only 5 iterations: 


Iteration 

f(x) 

11g(x)11 

0 

6.134640055 

9.4126587853e+00 

1 

1.976896063 

2.2780206611e-01 

2 

1.969726543 

2.6269970505e-03 

3 

1.969725575 

3.6873326702e-07 

4 

1.969725575 

7.2224253099e-15 


Optimum solution = 
0.1125 
1.1247 


Iterations = 4 

The code of the Python’s script MinEx2WithPureNewton.py is: 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 


import numpy as np 

def LineSearch(f, q, x, p): 

a, b = 1-2/(1+np.sqrt(5)), 2/(1+np.sqrt(5)) 
alpha = 1.0 

while f(x+alpha*p) > f(x) + a*alpha*np.dot(g(x), p): 
alpha *= b 
return alpha 

def PureNewton(f, q, xO, Eps): 

X = xO ; 

Iterations = 0 ; 

print ( ' - ' ) 

print ( ' Iteration\t f(x)\t ||g(x)||') 

print ( ' - ' ) 

while np.linalg.norm(g(x), 2) > Eps: 
p = -np.linalg.solve(H(x), g(x)) 
alpha = LineSearch(f, q, x, p) 

X = X + alpha * p 

print{'{0:5.0f}'. format (Iterations), ' \ t\t ' , ... 

’ {0: 12 . lOf} ’ . format (f(x)) ,\ 

'\t', '{0 :10.8e}'. format (np.linalg.norm(g(x)))) 

Iterations += 1 

print ( '-' ) 

return x, Iterations 

f = lambda x: (10*x[0]**2 + x[l]**2)/2 + ... 

5*np.log(1+np.exp(-x[0]-x[1])) 
g = lambda x: np.array([10*x[0] - 5*np.exp(-x[0] - ... 

x[l] ) / (np.exp(-x[0] - x[l]) + 1), 
x[l] - 5*np.exp(-X[0] - x[1])/(np.exp(-x[0] - x[l]) + 1)]) 
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29 Den = lambda x: 1 / (2 *np. exp (x [ 0 ] + x[l]) + np. exp (2 *x [ 0 ] + ... 

2*x[l] ) + 1) 

30 H = lambda x: Den(x) * np.array ( [[5*(5*np.exp(x[0] + x[l]) + ... 

2*np.exp(2 *x[0] +\ 

31 2*x[l]) + 2), 5*np.exp (x[0] + x[l])], [5*np.exp(x[0] + x[l]), \ 

32 7*np.exp(x[0] + x[l]) + np.exp{2*x[0] + 2*x[l]) + 1]]) 

33 xO = np.array([l, 1]) 

34 Eps = le-8 

35 X, Iterations = PureNewton(f, g, xO, Eps) 

36 print ( ' X = ' , x) 

37 print (' Iterations = Iterations) 


10.1.4 Quasi Newton’s Methods 

The pure Newton’s method for solving unconstrained minimization problems 
suffer the problem that the inverse of the Hessian matrix must be evaluated 
at iteration k to find the search direction When the Hessian matrix is ill- 
conditioned the resulting search direction will be inaccurate and the iteration 
process could be unsuccessful. 

Instead of directly inverting the Hessian matrix, the quasi-Newton’s meth¬ 
ods look for approximations to the Hessian matrix in the different iterations. 

Therefore, is replaced by a positive definite matrix and the 

search direction at iteration k is This iterative process usu- 

ally starts from B^^'^ = I where I G is the identity matrix of type nxn. 

The quasi Newton methods differ in the way by which matrix B^^'^ is updated 
at iteration k, and following from that the research direction is computed. The 
most famous quasi Newton’s methods are the Davidon-Fletcher-Powell (DFP) 
and the Broyden-Flethcher-Goldfarb-Shanno (BFGS) methods [19, 3, 20]. 


10.1.4.1 The Broyden-Fletcher-Goldfarb-Shanno (BFGS) Method 

The BFGS statrs from initial approximation of the Hessian matrix « I. 
At iteration k, the approximate Hessian matrix is updated to obtain 

fjik+i) -^iiich will be used in the next iteration [20]. This is done as follows. 
At iteration k given =g{x^^^) and The procedure starts with 

finding the search direction p^^'^ by solving the linear system ff 
and the step size from the line search algorithm.Then two vectors are 
computed: and — g{x^^'^). Finally, the 

Hessian matrix is updated as: 




yik) (^yik)^ j 


(ywy 




i{k)y 
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The Python code MinWithBFGS.py is used to find the optimal solution of 
example 10.1 based on the BFGS algorithm: 


1 

import numpy as np 


3 

def LineSearch(f, g, x, p): 


4 

a, b = 1-2/(l+np.sqrt(5)), 2/{1+np.sqrt{5)) 


5 

alpha = 1.0 


6 

while f(x+alpha*p) > f(x) + a*alpha*np.dot(g(x), p^ 


7 

alpha *= b 


8 

return alpha 


10 

def BFGS(f, q, xO, Eps): 


11 

X = xO ; 


12 

Iterations = 0 ; 


13 

14 

• / 1 

' ) 

print ( ' Iteration\t f(x)\t\t ||g (x)||' ) 

15 

16 

• / 1 

' ) 

H = np.eye( len (xO), dtype=float) 

17 

while np.linalg.norm(g(x), 2) > Eps: 


18 

p = -np.linalg.solve(H, g(x)) 


19 

alpha = LineSearch(f, q, x, p) 


20 

s = alpha * p 


21 

y = g(x+alpha*s) - g(x) 


22 

X = X + s 


23 

H = H + np.outer(y, y)/np.inner(y,s)-(H@np.outer(s, 


24 

s)@H.T)/(s.T@H@s) 

print ('{0: 5 .Of} '. format (Iterations) , '\t 'r ■■■ 


25 

'{0:12.lOf}' . format (f(x)),\ 

'\t', '{0:10.8e}'. format (np.linalg.norm(g(x)))) 


26 

Iterations += 1 


27 

28 

• / 1 

' ) 

return x, Iterations 

29 

30 

f = lambda x: {10*x[0]**2 + x[l]**2)/2 + ... 


31 

5*np.log(1+np.exp{-x[0]-x[1])) 
g = lambda x: np.array([10*x[0] - 5*np.exp(-x[0] - 


32 

x[l] ) / (np,exp(-x[0] - x[l]) + 1), 
x[l] - 5*np.exp (-X [0] - x[1])/(np.exp (-x[0] - x[l]) 

+ 1) ]) 

33 

xO = np.array([1, 1]) 


34 

Eps = le-8 


35 

X, Iterations = BFGS(f, q, xO, Eps) 


36 

print ( ' X = ' , x) 


37 

print (' Iterations = Iterations) 



By executing the code, the optimal solution is obtained in 13 iterations: 


Iteration f(x) 


IIg(x)II 


0 

1 

2 


1.9971128385 

1.9842525656 

1.9703729526 


3.93916248e-01 

2.84906636e-01 

8.38711419e-02 
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10 

1.9697255747 

4.32879304e-08 

11 

1.9697255747 

3.93521430e-08 

12 

1.9697255747 

1.62355225e-10 


X = [0.11246719 1.12467185] 

Iterations = 13 

The corresponding MATLAB code is: 


1 

ciear ; clc ; 


2 

f = 

@ (x) 5 *x(l)"2 + x(2)''2/2 + 5*log ( 1+exp (-X (1) - 

-x(2))) ; 

3 

g = 

@ (x) [10*x(l) - 5*exp(-x(l) - X (2) ) / (exp (-X (1) 

- x(2) ) +1) ; 

4 

x(2) 

- 5*exp(-x(l) - X (2) ) / (exp {-X (1) - x(2)) + 1) 

] ; 

5 

xO = 

[1.0; 1.0] ; Eps = le-8 ; 


6 

[X, 

Iterations] = BFGS(f, q, xO, Eps) ; 


7 

disp (’ Optimum solution = ') ; disp(x) ; 


8 

disp ([' Iterations = ’ num2str (Iterations)]) ; 


10 

function [x, Iterations] = BFGS(f, q, xO, Eps) 


11 


X = xO ; 


12 


Iterations = 0 ; 


13 


fprintf ( ' - 

— -\n') ; 

14 


fprintf ( ' Iteration\t\t f (x) \t\t\t ||g(x)||\n') 

; 

15 


fprintf ( '- 

- —\n') ; 

16 


fprintf (' %5i\t\t%14 . 9f\t\t%12 . 10e\n ' , Iterations, f{xO), ... 



norm (g(xO), 2)) ; 


17 


H = eye(length (xO)) ; 


18 


while norm(g(x), 2) > Eps 


19 


p = -H\g(x) ; 


20 


alpha = LineSearch(f, q, x, p) ; 


21 


s = alpha*p ; 


22 


y = g (x+s) -g (x) ; 


23 


X = X + alpha*p ; 


24 


H = H + y*y '/ (y '*s)-H*{s*s')*H'/(s'*H*s) 


25 


Iterations = Iterations + 1 ; 


26 


fprintf (' %5i\t\t%14 . 9f\t\t%12 . 10e\n ' , Iterations, ... 



f (x) , norm(g(x), 2)) ; 


27 


end 


28 


fprintf ( '- 

—-\n') ; 

29 

end 



30 




31 

function alpha = LineSearch(f, q, x, p) 


32 


a = 1-2/ (l+sqrt (5)) ; 


33 


b = 2 / (l+sqrt (5)) ; 


34 


alpha =1.0 ; 


35 


while f (x+alpha*p) > f(x) + a*alpha*g(x) '*p 


36 


alpha = b*alpha ; 


37 


end 


38 

end 
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10.1.4.2 The Davidon-Fletcher-Powell (DFP) Algorithm 

The Davidon-Fletcher-Powell (DFP) method has a close form as the BFGS to 
update the approximate Hessian matrix. At iteration k, the vectors and 
are computed the same way as in the BFGS algorithm. When formulat- 
ing the Hessian matrix vectors and of the BFGS method 

exchange their role in the DFP method [3, 20]. The formula of updating the 
Hessian matrix at iteration k is: 


s(G fs(fe)) 

jjik+l) ^ jj(k) \ J 


{y(k))^ H^k)y{k) 


Example 10.2 In this example, the DFP method will be implemented to 
minimize the quadratic function 


f{xi,X2,X3) = {xi-2)‘^ + {l - X^if + X 2 + I 


The Python code is: 


1 import numpy as np 

2 

3 def LineSearch(f, g, x, p): 

4 a, b = 1-2/(1+np.sqrt(5)), 2/{1+np.sqrt{5)) 

5 alpha = 1.0 

6 while f(x+alpha*p) > f(x) + a*alpha*np.dot(g(x), p): 

7 alpha *= b 

8 return alpha 

9 

10 def DFP(f, q, xO, Eps): 

11 X = xO ; 

12 Iterations = 0 ; 

13 print ( ' - ' ) 

14 print ( ' Iteration\t f(x)\t\t | |g (x) | | ' ) 

15 print ( '-' ) 

16 H = np.eye( len (xO), dtype=float) 

17 while np.linalg.norm(g(x), 2) > Eps: 

18 p = -np.linalg.solve(H, g(x)) 

19 alpha = LineSearch(f, q, x, p) 

20 s = alpha * p 

21 y = g(x+alpha*s) - g(x) 

22 X = X + s 

23 H = H + np.outer(s, s)/np.inner(y,s)-(H@np.outer(y, ... 

y)@H.T)/(y.T@H@y) 

24 print (' {0: 5 .Of} format (Iterations), '\t ... 

'{0:12.10f}' . format (f(x)) 

25 '\t’, '{0:10.8e}'. format (np.linalg.norm(g(x)))) 

26 Iterations += 1 

27 print ( '-' ) 

28 return x, Iterations 

29 

30 f = lambda x: {{x[0]-2)**2+{1+x[2])**2)+{1+x[1]**2) 

31 g = lambda x: np.array([2*x[0] - 4 , 2*x[l], 2*x[2] + 2]) 
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32 xO = np.array([l, 1 , 1]) 

33 Eps = le-8 

34 X, Iterations = DFP(f, q, xO, Eps) 

35 print ( ' X = ' , x) 

36 print (’ Iterations = Iterations) 


By executing the code, the minimization problem is solved in 8 iterations: 


Iteration f(x) 

11g(x)11 

0 

1.3343685400 

1.15649218e+00 

1 

1.0010384216 

6.44491002e-02 

2 

1.0000032249 

3.59162526e-03 

3 

1.0000000100 

2.00154416e-04 

4 

1.0000000000 

1.11542233e-05 

5 

1.0000000000 

6.21603559e-07 

6 

1.0000000000 

3.46407792e-08 

7 

1.0000000000 

1.93046439e-09 


X = [ 2.00000000e+00 3 .94054416e-10 -9.99999999e-01] 

Iterations = 8 

The MATLAB code is: 


1 ciear ; clc ; 

2 f = @(x) (x(l)-2)'2 + (l+x(3))^2 + x(2)"2 + l ; 

3 g = @(x) [2*{x{l)-2); 2*x{2); 2*(x(3)+l)] ; 

4 xO = [1.0; 1.0; 1] ; Eps = le-8 ; 

5 [x, Iterations] = DFP (f, g, xO, Eps) ; 

6 disp (’ Optimum solution = ') ; format long e ; disp(x) ; 

7 disp ([’ Iterations = ' num2str (Iterations)]) ; 

8 

9 function [x, Iterations] = DFP(f, q, xO, Eps) 

10 X = xO ; 

11 Iterations = 0 ; 

12 fprintf ( '- \n ' ) ; 

13 fprintf (' Iteration\t\t f (x) \t\t\t ||g{x)||\n') ; 

14 fprintf ( ' -\n ' ) ; 

15 fprintf (' %5i\t\t%14 . 9f\t\t%12 . 10e\n ' , Iterations, f{x0), ... 

norm (g(xO), 2)) ; 

16 H = eye (length (xO) ) ; 

17 while norm(g(x), 2) > Eps 

18 p = -H\g(x) ; 

19 alpha = LineSearch(f, g, x, p) ; 

20 s = alpha*p ; 

21 y = g(x+s)-g(x) ; 

22 X = X + alpha*p ; 

23 H = H + s*s'/(y'*s)-H*{y*y')*H'/(y'*H*y) ; 

Iterations = Iterations + 1 ; 


24 













274 


Solving Optimization Problems: Nonlinear Programming 


25 fprintf ( ' %5i\t\t%14 . 9f \t\t%12 . 10e\n ' , Iterations, ... 

f (x) , norm(g(x), 2)) ; 

26 end 

27 fprintf ( '- \n ' ) ; 

28 end 

29 

30 function alpha = LineSearch(f, g, x, p) 

31 a = 1-2/(1 + sqrt (5) ) ; 

32 b = 2/(1 + sqrt (5) ) ; 

33 alpha = 1.0 ; 

34 while f (x+alpha*p) > f(x) + a*alpha*g(x)'*p 

35 alpha = b*alpha ; 

36 end 

37 end 


10.1.5 Solving Unconstrained Optimization Problems with 
MATLAB 

The MATLAB function fminunc is used to solve the unconstrained minimiza- 
tion problem: 

min f(x),x G R"" 

The function fminunc receives mainly an objective function / and a starting 
point xq and optionally a parameter options to return mainly the optimum 
solution X and optionally the value of the objective function at the opti- 
mal solution and other optional outputs. Through the parameter options 
the user can customize the parameters of the optimization process such as 
the algorithm, maximum number of iteration, maximum number of function 
evaluations, the tolerance in the value of the objective function, etc. 

Example 10.3 In this example the MATLAB function fminunc will be used 
to solve the unconstrained minimization problem: 

. I0x‘i+xl -n-To 

min- - -^ + 51oge ^ ^ 

2 

starting from the point 


>> f = @(x) 5 *x(l)''2 + x(2)''2/2 + 5*log(l+exp(-x(l)-x(2))) ; 
» xO = [1; 1] ; 

>> Options = optimsetC^Display^ ^Iter\ 'TolFun’, le-7) ; 


V 

V 

II 

fminunc (f, [1; 1], 

Options) 



First-order 




Iteration Func-count 

f(x) 

Step-size 

optimality 

0 

3 

6.13464 


9.4 

1 

6 

2.08295 

0.106338 

1.39 

2 

9 

1.98574 

1 

0.238 

3 

12 

1.98063 

1 

0.194 

4 

15 

1.96973 

1 

0.00244 
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5 

18 

1.96973 

1 

4.72e-05 

6 

21 

1.96973 

1 

1.67e-06 

7 

24 

1.96973 

1 

2.98e-08 


Local minimum found. 

Optimization completed because the size of the gradient is less than 
the selected value of the optimality tolerance. 

<stopping criteria details> 

X = 

0.1125 

1.1247 

10.1.6 Solving an Unconstrained Problem with Python 

Python has many functions to solve the unconstrained minimization prob¬ 
lems. Those solvers include the functions fmin, fmin_bfgs, fmin_ncg and 
minimize. 

To solve the problem of example 10.3 with Python, the following Python 
instructions can be used: 

In [1]: import numpy as np 

In [2]: from scipy.optimize import fmin 

In [3]: xO = np. array ( [1. , 1.]) 

In [4]: f = lambda x: (10*x[0]**2 + x[l]**2)/2 + 5*np.log(l+np.exp(-x[0]-x[1])) 
In [5]: Eps = le-8 

In [6]: xopt = fmin(f, xO, disp=l, xtol=Eps) 

Optimization terminated successfully. 

Current function value: 1.969726 
Iterations: 65 
Function evaluations: 126 
In [7]: print(xopt) 

[0.11246719 1.12467185] 

In [8]: from scipy.optimize import fmin_bfgs 

In [9]: g = lambda x: np.array([10*x[0] - 5*np.exp(-x[0] - x[l])/\ 

(np.exp(-x[0] “ x[l]) + 1), x[l] - 5*np.exp(-x[0] - x[l])/\ 

(np.exp(-x[0] - x[l]) + 1)]) 

In [10]: xopt = fmin_bfgs(f, xO, fprime=g, disp=l) 

Optimization terminated successfully. 

Current function value: 1.969726 
Iterations: 7 
Function evaluations: 8 
Gradient evaluations: 8 
In [11]: print(xopt) 

[0.11246715 1.1246719 ] 

In [12]: from scipy.optimize import fmin_ncg 

In [13]: Den = launbda x: 1/(2*np. exp(x [0] + x[l]) + np. exp(2*x [0] + 2*x[l]) + 1) 
In [14]: H = lambda x: Den(x) * np.array([[5*(5*np.exp(x[0] \ 

+ x[l]) + 2*np.exp(2*x[0] + 2*x[l]) + 2), 5*np. exp(x [0] + x[l])],\ 
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[5*np.exp(x[0] + x[l]), 7*iip.exp(x[0] + x[l]) + iip.exp(2*x[0] + 2*x[l]) + 1]]) 
In [15]: xopt = finin_ncg(f, xO, fprinie=g, fhess=H, disp=l) 

Optimization terminated successfully. 

Current function value: 1.969726 
Iteratione: 4 
Function evaluations: 5 
Gradient evaluations: 8 
Hessian evaluations: 4 
In [16]: print(xopt) 

[0.11246719 1.12467185] 

In [17]: Sol = minimize(f, xO) 

In [18]: print(Sol) 

fun: 1.969725574672448 

hess_inv: array([[ 0.09705991, -0.04667767], 

[-0.04667767, 0.55809497]]) 

j ac: array([-2.98023224e-07, 5.96046448e-08]) 

message: ’Optimization terminated successfully.^ 

nfev: 32 

nit: 7 

njev: 8 

status: 0 

success: True 

x: arrayC[0.11246715, 1.1246719 ]) 


10.1.7 Solving Unconstrained Optimization Problems with 
Gekko 

In Gekko the optimization problems can be solved at mode 3. In this sec- 
tion two unconstrained optimization problems. The two problems are taken 
from the Mathworks website https;//www.mathworks. com/help/optim/ug/ 
fminunc. html#butpb7p-4. The problems are given in two following examples. 


Example 10.4 In this example Gekko will be used to solve the unconstrained 
minimization problem: 

min f{xi,X 2 ) = 3a;f + 2 xiX 2 + X 2 — Ixi + 5x2 

a;eR2 

The code is embedded on the Python script MinExluncWithGekko. py 


1 from gekko import GEKKO 

2 m = GEKKO() 

3 xl = m.Var(1) 

4 x2 = m.Var(1) 

5 

6 m.Obj(3*xl**2 + 2*xl*x2 + x2**2 - 4*xl + 5*x2) 

7 m.options.IMODE = 3 

8 m.solve() 

9 print ( ' Solution found at x = ') 
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10 print ('{0:12.lOe}' . format (xl[0])) 

11 print ('{0:12.lOe}' . format (x2[0])) 

By running the code, the following results are obtained: 

EXIT: Optimal Solution Found. 

The solution was found. 

The final value of the objective function is -16.3750000000000 


Solver : IPOPT (v3.12) 

Solution time : 4.699999990407377E-003 sec 

Objective : -16.3750000000000 

Successful solution 


Solution found at x = 

2.2500000000e+00 

-4.7500000000e+00 

Example 10.5 The unconstrained optimization problem is to minimize 
min 100 (a ;2 + (1 ~ 2 : 1 )^ 


The Python script MinEx2uncWithGekko. py solves the problem: 


1 

from gekko import GEKKO 


2 

m = GEKKO ( ) 


3 

xl = m.Var(-1.) 


4 

x2 = m.Var(2.) 


6 

m.Obj(100*(x2 - xl**2)**2 + 

(l-xl)**2) 

7 

m.options.IMODE = 3 


8 

m .solve () 


9 

print (’ Solution found at x = 

') 

10 

print ('{0:12.lOe} ' . format {xl 

[0] ) ) 

11 

print ('{0:12.lOe} ' . format {x2 

[0] ) ) 


Executing the code gives the following results: 


EXIT: Optimal Solution Found. 
The solution was found. 


The final value of the objective function is 
7.912533725331136E-015 
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Solver ; IPOPT (v3.12) 

Solutiori time : 1.490000000922009E-002 sec 

Objective : 7.912533725331136E-015 

Successful solutiori 


Solution found at x = 
9.9999991265e-01 
9.9999982362e-01 


10.2 Solving Constrained Optimization Problems 

At the beginning an optimization problem with equality constraints is consid- 
ered. The form of the problem is: 

min/(ai), a; € R” (10.8) 

subject to the equality constraints: 

E{x)^0 (10.9) 


where E € R^, p<n. 

To derive the necessary and sufficient optimality conditions, construet the 
Lagrangian system: 

L{x) = f{x) + E^{x)X (10.10) 

with A S is the vector of Lagrange multipliers. Then, the first-order nec¬ 
essary conditions for (a;*,A*) to be optimal are: 

r Va^L{x*,X*) = 0 = Va^f{x*) + Va^E^{x*)X* 

{ ( 10 . 11 ) 

[ VxL{x*,X*) = 0 = E{x*) 


where WxE{x) is the Jacobian matrix evaluated at the point x G R"". 

The second-order necessary condition is: 

VlMx*,X*) = Vl^fix*) + J2yj^LE{x*)=H{x*)GR^ ( 10 . 12 ) 

i=i 


be positive semi-definite. 
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The second-order sufficient condition for (a;*,A*) to be optimal is: 

p 

= + = (10.13) 

i=i 


be positive-definite. 

If VxE^{x) G is a full-rank matrix, it has a QR factorization of the 

form: 

V£;^(x) = [Q z] 

where Q £ Z G R g EP^p and 0 is a matrix of zeros of type 

(n—p) X p. The matrix Q is a basis for the column space of VxE^{x), whereas 
.Z is a basis for its null space. 

At the optimum solution x*, the gradient of the objective function is 
orthogonal to the constraints surface. That is the projection of the gra¬ 
dient vector onto the constraint surface is zero. This can be expressed as 
z* V/(a;*) = 0, which is equivalent to the first order necessary conditions 
(10.11).The second order necessary conditions for optimality of (a;*, A*) that 
are equivalent to (10.12) is Z*^H*Z* be positive semi-definite, and the sec¬ 
ond order sufficient condition of optimality of (a;*, A*) that is equivalent to 
(10.13) is that Z*"^H*Z* be positive definite [20]. 

The Newoton’s optimization methods iterate to find a couple (a:*, A*) £ 
■^(n-p)x{n-p) necessary conditions (10.11) and (10.12) are ful- 

filled. At iteration k, the Karush-Kuhn-Tucker (KKT) system is composed from 
a previously computed couple to find (a;*^^+^),A*-^"*"^^) by solving 

the KKT System: 


R 

0 


ff(a:(*)) 

(ve'^{x<^'^'>)Y 


- s(fc) ■ 


■ -V/(a;W) ■ 

. V£;^(a;W) 

0 


;^(fe+i) 


-£;(a;W) 


where — x^^'^. 

Now, if considered a minimization problem with inequality constraints of 
the form: 


min /(a;), / : R’^ ^ R, a; £ R" 

E{x) = 0, E-.R^ 

I{x)<Q, 

Now, if & £ R” is a feasible point, it is called regular if the columns of 
V/^(a;) are linearly independent in R'^. At a regular point x an equality or 
inequality constraint c(a;) is called active if c(ai) = 0, otherwise it is inactive at 
X. Let A be the set containing all the active constraints at x, then A is called 
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the active set at x. At the optimal solution x* all the equality constraints 
shall satisfy: 

E,{x*) = 0, j = l,...,p. 

The inequality constraints are divided between active and inactive. Let S* 
be the indices set of inequality constraints {j : Ij{x*) = 0, j G {l,...,g}} 
Therefore, the active set A* at the optimal solution x* consists of E{x*)U 
{I,{x*):jeS*}. 

The Lagrangian system of the constrained minimization problem (10.1)- 
(10.3) is defined by: 

L{x,\pb) = f{x) + E'^{x)X + l'^{x)fj,, (10.15) 

where XgMP and /x G . 

Then, the first-order necessary conditions of optimality for a point 
{x*,X*,pi*) are: 

W^L{xb*,X*,n*) = 0 = Wf{x*) + WE'^{x*)X*+Wl'^{x*)n* (10.16) 


VxL{xb*,X\pi*) = t) = E{x*) (10.17) 

W^L{xb*,X*,pi*) = l{x*)<0, (10.18) 

fi* > 0,j = l,...,q (10.19) 

fi*Ij{x*) = = ( 10 . 20 ) 

Condition (10.20) is called the complementary slack condition which means 
that /.i* > 0 if j G S* {Ij{x*) is active) and otherwise. 

The second order necessary optimality condition is that: 

yW^a,L{x*,X*,pi*)y ( 10 . 21 ) 

be positive semi-definite. 

The second order sufficient optimality condition is: 

yW^a,L{x*,X*,y*)y ( 10 . 22 ) 


be positive definite. 

10.2.1 Solving Constrained Optimization Problems with 
MATLAB fmincon Fnnction 

The MATLAB function fmincon is used for solving constrained minimization 
problems. It receives an objective function f{x), a starting point x^, a function 
handle to nonlinear constraints function, the optimization process options. If 
linear equality and inequality constraints present, they can be passed to the 
function fmincon as well, so are the variables bounds. It receives mainly the 
solution of the problem and optionally the value of the objective function 
at the optimal solution, the exit flag, the Lagrange multipliers, gradient and 
Hessian matrix at the optimum point [17]. 
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The form of implementing the fmincon solver is: 

>> [xopt, fmin, exFlg, Lamd, Grad, Hess] = fmincon(@f, xO, Aiq, 
biq, ... Aeq, beq, lowbnd, upbnd, Onlcon, Options) ; 

Example 10.6 In this example, the function fmincon will be used for solving 
the constrained minimization problem: 

min a:iX 4 (a;i + a :2 + a: 3 )+a :3 

subject to inequality constraint: 

xi ■ X 2 ■ X 3 ■ Xi> 25 


and equality constraint: 

x‘l+X 2 +x‘^ + x‘l = 40 


where 


— 1 < Xi,X 2 ,X 3 ,X 4 < 5 


The MATLAB script MinExlWithfmincon.m solves the optimization problem 
using the fmincon function: 


1 

% MinExlWithfmincon.m 


2 

ciear ; clc ; 


3 

xO = [1; 5; 5; 1]; % Make a starting guess at 

the solution 

4 

Algorithms = ["sqp", "active-set", "interior-point 

", ... 


"sqp-legacy"] ; 


5 

options = optimoptions(@fmincon, ’ Algorit hm' , ' sqp ' 

, ' Disp ' , ... 


' Iter ' , ' PlotFcns ' , ' optimplotfval ' ) ; 


6 

[x,fval] = fmincon(@obj fun,xO, [], [], [], [], [], [],@confun,options); 

7 

fprintf ( ’ Optimal solution found at \nx ... 



=\n%12.6e\n%12.6e\n%12.6e\n%12.6e\n\n', x{l), 
x(4)) ; 

x(2) , x(3) , ... 

8 

fprintf ( ’Value of objective function = %10.7f\n\n' 

, fval) ; 

10 

function f = objfun(x) 


11 

f = X (1) *x (4) * (X (1) +x (2) +x (3) ) +x (3) ; 


12 

end 


13 



14 

function [ic, ec] = confun(x) 


15 

% Nonlinear inequality constraints 


16 

ic(l) = -X (1) *x (2) *x (3) *x (4) +25.0 ; 


17 

ic(2) = -x(l) + 1 ; 


18 

ic(3) = x(l) - 5 ; 


19 

ic(4) = -x(2) + 1 ; 


20 

ic(5) = x{2) - 5 ; 


21 

ic(6) = -x(3) + 1; 


22 

ic(7) = x(3) - 5; 


23 

ic(8) = -x(4) + 1; 


24 

ic(9) = x(4) - 5 ; 


25 

% Nonlinear equality constraints 


26 

ec = -sum(x.*x) + 40 ; 


27 

end 
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Running the MATLAB code gives: 


Iter Func-count fval 

step optimality 

Feasibility 

Step Length 

Norm of 

Flrst-order 

0 

5 

1.600000e+01 

1.200e+01 

l.OOOe+00 

O.OOOe+00 

1.200e+01 

1 

10 

1.606250e+01 

1.387e+00 

l.OOOe+00 

1.159e+00 

2.077e+00 

2 

15 

1.696396e+01 

8.077e-02 

l.OOOe+00 

1.875e-01 

1.347e-01 

3 

20 

1.701372e+01 

4.614e-04 

l.OOOe+00 

1.682e-02 

1.317e-02 

4 

25 

1.701402e+01 

8.243e-08 

l.OOOe+00 

2.871e-04 

5.985e-05 

5 

30 

1.701402e+01 

3.579e-ll 

l.OOOe+00 

5.956e-06 

2.446e-07 


Local minimum found that satisfies the constraints. 

Optimization completed because the objective function is non-decreasing in 
feasible directions, to within the default value of the optimality tolerance, 
and constraints are satisfied to within the default value of the constraint 
tolerance. 

<stopping criteria details> 

Qptimal solution found at 

X = 

l.OOOOOOe+00 

4.743000e+00 

3.821150e+00 

1.379408e+00 


Value of objective function = 17.0140173 


Example 10.7 In this example, the fmincon will use the algorithms sqp, 
interior-point, active-set and sqp-legacy to solve the constrained min- 
imization problem: 


. /2 3\2 

min X 2 {x-i — Xo) 


4 

X\X2 


subject to: 


—(!)<xi+X2+x^< 6 , 


and 


-1<X2< 1. 


The MATLAB script MinEx2Withfmincon.m solves the above minimization 
proplem. Its code is: 


1 % MinEx2Withfmincon .m 

2 ciear ; clc ; 

3 xO = [-1;-1; -1]; % Make a starting guess at the solution 

4 Algorithms = ["sqp", "active-set", "interior-point", ... 

"sqp-legacy"] ; 

5 

6 for n = 1 : length (Algorithms) 

7 options = optimoptions(@fmincon, ' Algorithm ', Algorithms(n), 

' Display ' , ' off ’ ); 
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8 fprintf ( ' Solution with %s algorithm : \n ' , Algorithms(n)) ; 

9 [x,fval] = ... 

fmincon(@obj fun,xO, [], [], [], [], [], [],@confun,options); 

10 fprintf (' Optimal solution found at \nx ... 

=\n%12 . 6e\n%12 . 6e\n%12 . 6e\n ' , x(l), x{2), x(3)) ; 

11 fprintf (' Value of objective function = %10.7f\n\n', fval) ; 

12 end 

13 

14 function f = objfun(x) 

15 f = X (2) * (l + 2*x (1) ^2 - x{3)"3)"2 + 4 / {x (1) *x (2 ) ) ; 

16 end 

17 

18 function [ic, ec] = confun(x) 

19 % Nonlinear inequality constraints 

20 ic(l) = -x(l) -x(2) -x{3)+6 ; 

21 ic(2) = x{l) + x(2) + x{3) - 6; 

22 ic(3) = -x{2) + 1 ; 

23 ic (4) = X (2) - 1 ; 

24 % Nonlinear equality constraints 

25 ec = [ ] ; 

26 end 


Running the code gives the following results: 

Solution with sqp algorithm; 

Optimal solution found at 

X = 

2.575360e+00 

l.OOOOOOe+00 

2.424640e+00 

Value of objective function = 1.5532974 

Solution with active-set algorithm; 

Optimal solution found at 

X = 

2.575360e+00 

l.OOOOOOe+00 

2.424640e+00 

Value of objective function = 1.5532974 

Solution with interior-point algorithm: 

> In backsolveSys 
In soIveKKTsystem 
In computeTrialStep 
In barrier 

In fmincon (line 800) 

In minexSwithfmincon (line 8) 

Warning; Matrix is singular to working precision. 
Optimal solution found at 
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X = 

2.575362e+00 

l.OOOOOOe+00 

2.424638e+00 

Value of objective function = 1.5532974 

Solutiori with sqp-legacy algorithm: 

Optimal solutiori found at 

X = 

2.575360e+00 

l.OOOOOOe+00 

2.424640e+00 

Value of objective function = 1.5532974 

10.2.2 Solving Constrained Minimization Problems in Python 

Constrained minimization problems are solved in Python with the function 
minimize located in the library scipy. optimize. It receives the objective 
function, an initial starting point and the options of the optimization process. 
It returns the optimum solution and the value of the objective function at the 
optimal solution [18]. 

In this section Examples 10.6 and 10.7 will be solved using Python Scripts 
MinExlminimize .py and MinEx2minimize .py, respectively. 

The Python script MinExlWithminimize. py solves Example 10.6. Its 
code is: 


1 # MinExlWithminimize 

2 import numpy as np 

3 from scipy.optimize import minimize 

4 

5 def objfun (x) : 

6 return x[0]*x[3]*(x[0]+x[l]+x[2])+x[2] 

7 

8 def ic(x): 

9 return x [0]*x[1]*x[2]*x[3]-25.0 
10 

11 def ec (x) : 

12 return 40.0 - sum(x*x) 

13 

14 xO = np.array{[1., 5., 5., 1.]) 

15 

16 cons = [{'type': 'ineq', 'fun': ic}, {'type': 'eq', ' fun ' : ec}] 

17 lubs = [(1,0, 5.0), (1.0, 5.0), (1.0, 5.0), (1.0, 5,0)] 

18 Sol = minimize(objfun,xO, method= ' SLSQP ' , bounds=lubs, ... 

constraints=cons) 

19 X = Sol. X 

20 fval = Sol.fun 

21 print (' Solution found at x = ') 

22 print ('{0:12.lOe}' . format (x[0])) 

23 print ('{0:12.lOe}' . format (x[1])) 
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24 print ('{0:12.lOe}' . format (x[2])) 

25 print ('{0:12.lOe}' . format (x[3])) 

26 print ('Value of the Objective function = ' + ... 

str(('{0:10.7f}'. format (fval)))) 

Executing the code gives: 

runfile(’D;/PyFiles/MinExlWithminimize.py’, wdir=’D;/PyFiles’) 
Solution found at x = 
l.OOOOOOOOOOe+00 
4.7429960656e+00 
3.8211546642e+00 
1.3794076394e+00 

Value of the Objective function = 17.0140172 

To solve Example 10.7 with Python, the script MinEx2Withminimize. py 
implements the solution: 


1 

import numpy as np 


2 

from scipy.optimize import minimize 


4 

def objfun (x) : 


5 

return 4*x[1]*(1+2*x[0]- x[2]**3)**2 

+ x[0] /x[l] 

7 

def ic (x) : 


8 

icon = np.zeros(4) 


9 

icon[0] = -x[0] - x[l]**2 - x[2] + 6 


10 

icon[l] = x[0] + x[l]**2 + x[2] - 6 


11 

icon[2] = -x[l] + 1 


12 

icon[3] = x[l] - 1 


13 

return icon 


14 



15 

xO = np.array([1., 1 ., 1.]) 


16 



17 

cons = [{'type': 'ineq', 'fun': ic}] 


18 

bnds = [(-np.inf, np.inf), (-1.0, 1.0), (-np 

inf, np.inf)] 

19 

OptSol = minimize(objfun,xO, method= ' SLSQP ' , 

bounds=bnds, . . . 


constraints=cons) 


20 

print (OptSol) 


21 

X = OptSol. X 


22 

print (' Solution found at x = ') 


23 

print ('{0:12.lOe}' . format (x[0])) 


24 

print ('{0:12.lOe}' . format (x[1])) 


25 

print ('{0:12.lOe}' . format (x[2])) 


26 

print ('Value of the Objective function = ' + 



str (('{0:10.7f}'. format (OptSol.fun)))) 



By executing the code, the following results are obtained: 


runfile(’D:/PyFiles/MinEx2Withininimize.py’, wdir=’D:/PyFiles’) 
fun: 2.5748936729216085 

jac: array([ 0.63140237, -2.57473353, 0.63150746]) 

message: 'Optimization terminated successfully.’ 
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nfev: 46 
nit; 8 
njev; 8 
status; 0 
success; True 

x; arrayC[2.57481362, 1. , 2.42518638]) 

Solutiori found at x = 

2.5748136179e+00 

l.OOOOOOOOOOfe+00 

2.4251863821e+00 

Value of the Objective function = 2.5748937 

10.2.3 Solving Constrained Optimization with Gekko Python 

Gekko Python Software solves constrained and unconstrained optimization 
problems at mode 3 solver [2]. To solve Example 10.6 with Gekko, the the 
script MinExlWithGekko. py can be used: 


1 # MinExlWithGekko . PY 

2 from gekko import GEKKO 

3 m = GEKKO ( ) 

4 xl = m.Var(l;. lb=l, ub=5) 

5 x2 = m.Var(5, lb=l, ub=5) 

6 x3 = m.Var(5, lb=l, ub=5) 

7 x4 = m.Var(l, lb=l, ub=5) 

8 m.Equation (xl * x2 * x3 * x4 > 25) 

9 m.Equation (xl**2+x2**2+x3**2+x4**2 == 40) 

10 m.Obj(xl * x4 * (xl + x2 + x3) + x3) 

11 m.options.IMODE = 3 

12 m. solve () 

13 print ( ’ Solution found at x = ') 

14 print ('{0:12.lOe}' . format (xl[0])) 

15 print ('{0:12.lOe}' . format (x2[0])) 

16 print ('{0:12.lOe}’ . format (x3[0])) 

17 print ('{0:12.lOe}' . format (x4[0])) 

Running the code gives the following results: 

EXIT; Optimal Solution Found. 

The solution was found. 

The final value of the objective function is 17.0140171270735 


Solution time ; 
Objective : 

Successful solution 


Solver 


IPOPT (v3.12) 

1.000000000931323E-002 sec 
17.0140171270735 






Solving Constrained Optimization Problems 


287 


Solutiori found at x = 

1.0000000570e+00 

4.7429996300e+00 

3.8211500283e+00 

1.3794081795e+00 

The Python script MinEx2WithGekko .py is used to solve Example 10.7. 


1 # MinEx2WithGekko . py 

2 from gekko import GEKKO 

3 m = GEKKO ( ) 

4 xl = m.Var(l, lb=0, ub=6.) 

5 x2 = m.Var(l, lb=-l., ub=6.) 

6 x3 = m.Var(l, lb=0, ub=6.) 

7 m.Equation(xl + x2**2 + x3 > -6) 

8 m.Equation(xl + x2**2 + x3 < 6) 


9 


10 m.Obj(x2*(l+2*xl**2 - x3**3)**2 + 4./(xl*x2)) 

11 m. options . IMODE = 3 

12 m. solve () 

13 print ( ' Solution found at x = ') 

14 print ('{0:12. lOe} ' . format (xl [0] ) ) 

15 print ('{0:12.lOe}' . format (x2[0])) 

16 print ('{0:12.lOe}' . format (x3[0])) 

17 fval = x2 [0] * (l + 2*xl [0] **2 - x3[0]**3)**2 + 4 . / (xl [ 0 ] *x2 [ 0 ] ) 

18 print ('Value of the Objective function ='+... 

str( ('{0:10. 7f}' . format (fval)))) 


Running the program gives the following results: 

EXIT: Optimal Solution Found. 

The solution was found. 

The final value of the objective function is 1.43580274955049 


IPOPT (v3.12) 

1.930000001448207E-002 sec 
1.43580274955049 


Solver 

Solution time 
Objective 


Successful solution 


Solution found at x = 

2.1288804479e+00 

1.3087777155e+00 

2.1582204319e+00 

Value of the Objective function = 1.4358027 







Taylor8i Francis 

Tayior & Francis Group 

http://taylorandfrancis.com 



11 


Solving Optimal Control Problems 


Abstract 

Optimal Control Problems (OCPs) represent a model for a wide variety of real- 
life phenomena. Some of these applications include the control of infectious 
diseases, the continuous stirred tank reactor (CSTR), biological populations, 
population harvesting, etc. 

This chapter presents the basies of optimal control. It then discusses the 
uses of indirect and direct transcription methods for solving them numerically 
using MATLAB® and Python. It also discusses the use of the gekko Python 
for solving the optimal control problems. 

The chapter consists of six sections, organized as follows. The statement of 
the problem is in the first section. The second and third sections discuss the 
necessary conditions for the optimal control. Ideas about some of numerical 
methods for solving the optimal control problems are presented in Section 4. 
Section 5 discusses the numerical solution of optimal control problems based 
on the indirect transcription methods. The numerical methods for solving opti¬ 
mal control problems based on the direct transcription methods are presented 
in Section 6. 


11.1 Introduction 

This chapter considers an optimal control problem (OCP) of the form: 

minimize (p(^x{tf))+ / LQ{t,x{t),u{t)) dt (H-l) 

«eu 

The dynamies is described by a System of ordinary differential equations 
defined on the interval 

x{t) = f {t,x{t),u{t)) ,\/t G [to,tf], (11.2) 

with initial condition 

x{to) = x^ (11-3) 
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The System might be subject to continuous state inequality constraints 
J(t,a;(t)) < 0, (11-4) 

and equality constraints 

E{t,x{t)) = Q, VtG[to,t/]. (11.5) 

It is subject to terminal conditions: 

= 0 , ( 11 . 6 ) 

where x{t) = [xi(t),• • • ,Xn(t)]^ G i?” is the state vector and u{t) = 

G is the control vector. 

The terminal time is a fixed positive real number. U is the class of 
all piecewise continuous functions u{t) defined on [tQ,tf), with ai < Ui{t) < 
bi, ai, bi G R. 

The function / : R x R"" x R”^ —>■ R"", is continuously differentiable with 
respect to each component of x and u, and piecewise continuous with respect 
to t. 

The term (p{x{tf)) in equation (11.1) is called the Mayer part, and the 
integral term of Equation (11.1) is called the Bolza paxt. A cost function 
could consist of only a Mayer part, only a Bolza part or a combination of 
both [10]. 


11.2 The First-Order Optimality Conditions and 
Existence of Optimal Control 

Notation: Throughout the chapter, we will use the following notations: 

Lo[t] = Lo{t,x{t),u{t)) and f[t] = f{t,x{t),u{t)). 

The Hamiltonian for problem (11.1)-(11.6) is given by 

n[t]^'H{t,x,u,p) f[t] 

then, the augmented cost function can be written as: 

f*-f 

J{u) = ip(tf,x{tf)) + v>'^'^{tf,x{tf))+ / [H[t]-p{t)'^x(t)]dt 

Jto 

The first-order necessary conditions for the optimality are found by apply- 
ing the variational principle to the augmented eost function, also referred to 
as Lagrangian [24, 11]. The augmented cost function is given by 

J{u) = Lp{tf,x{tf)) + v'^^{tf,x{tf))+ f [Lo[t]+p{t)'^{f[t]-x{t))]dt 

JtQ 

(11.7) 
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where p{t) G R" are the adjoint variables or the costate variables, n gR^ are 
Lagrange multipliers, the final time tf, may be fixed or free. 

The minimum of J{u) is determined by computing the variation SJ with 
respect to all the free variables and then equating it to the zero [15]. i. e. 
5J = 0. 


SJ = 


dip{tf,x{tf)) ^^ ^ dip{tf,x{tf)) 


^ , - OXf + U -—0tf+—^ cOXf 

dtf ^ dx{tf) ^ \dtf ■' dx{tf) ^ 

+ + (u[tf]-p^{tf)x{tf)^ Stf + 


/to 


an, an, an/ 

oa:+ —— du+ dp 


ax 


du 


ap 


where Sxf = 6x{tf) + x{tf)Stf. 

Integrating —p'^{t)Sx{t)dt by parts, gives 


dt 

( 11 . 8 ) 


'to 


—p^{t)5x{t)dt=—p^{tf)5x{tf)+ / p^{t)5x{t)dt 


(11.9) 


/to 


Inserting (11.9) into (11.8) and rearranging the terms we get 


ay} a^p 
atf~^ ax{tf) ‘ 




/to 


2^ [i|+p^(t) j ir. + (|t| j + (K [<l - i(i) 


atf ax{tf)‘ 

f an 


fan, 


Sp 


dt (11.10) 


The necessary optimality conditions define a stationary point, at which 
any arbitrary variations in the free variables, resuit in no change in the total 
cost [57, 54]. The first order optimality conditions are given by 


(i) equating the coeflicient of 5u to zero gives 

an. 


au 


[t] = 0, to<t<tf 


(11.11) 


(ii) equating the coefficient of 5x to zero gives the adjoint equations 


an 


P W = to<t<tf 

(iii) equating the coefficient of Sp to zeros gives the state equations 

Qnj 

x{t) = —[t] = f{t,x{t),u{t)), to<t<tf 


( 11 . 12 ) 


(11.13) 
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(iv) equating the coefRcient of 5x(tf) to zero, gives the transversality condi- 
tions 


P{tf) = 




u 


(11.14) 


dx{tf) dx{tf) 

(v) equating the coefficient of to zero, gives the terminal constraints 

^{tf,x{tf))=^ (11-15) 


(vi) equating the coefficient of Stf to zeros, gives the condition on the Hamil- 
tonian at the terminal time, 

''' at, at, 

Equations (11.11, 11.12, 11.13) define the Euler-Lagrange equations [54, 15] 
for the optimal control problems. 

Example 11.1 (Illustrative Example:) 


minimize J{u)=x{T) + a / u^{t)dt 

0^U^t)^Umax Jq 

subject to the dynamics: 

x(t) = ax{t) +u{t), a:(0) = xq 
The Hamiltonian of the optimal control problem is: 

7i{x{t),u{t),p{t)) = au^{t) +p{t) {ax{t) + u{t )), 


and the Euler-Lagrange equations are: 


fm 

du 

Pit) 

x{t) 


2au{t) +p{t) = 0 

ax{t) -|-M(t),a;(0) = xq 


in addition to the transversality condition 


P(T) 


dx(T) 

dx(T) 


If u*(t) and x*(t) are the optimal control and the resulting optimal trajectory 
and p*(t) is the co-state variable corresponding to the optimal trajectory. 
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then the tuple {u*{t),x*{t),p*{t)) can be found by solving the Euler-Lagrange 
equations as follows. 

By solving the second equation in p{t) together with the transversality 
condition, we get 

By solving the first equation in u{t), we get 


u*{t) 


P*{t) 

2a 


^a{T-t) 

2a 


Finally, by solving the third equation in x{t), we get 


x*{t) 



^a{T-t) 

4aa 


11.3 Necessary Conditions of the Discretized System 

We develop in this section a numerical method for solving an OCP without 
transforming it into one without time delays. 

Let N he a positive integer and h = then sq = to < si < • ■ ■ < skn = 
tf where Sj+i = Si + h is a partition for the interval Let x{si),p{si) 

and u{si) be the values of x{t),p{t) and u{t) at t = Si, i = 0,l,...,N. The 
Hamiltonian at Si is given by 

Ulsi] = 'H{si,x{si),x{si - T),p{si)) = Lo[si]+p'^{si)f[si] (H-H) 

The optimality conditions for the discretized problem are 

1 . 

dH 

— [si] = 0,So < Si < SiCAT (11.18) 

2 . 


p{si) = p{si+i) + h 
p{si) = p{si+i) + h 




dn 

dXr 


[si+i+r] , 


So < Si < S(^x-1)N 
S{K-i)N Si Si < Skn 

(11.19) 


3 . 


a5(si+i) = x{si) + h—[si] = x{si) + hf[si] 


so< Si< Skn (11.20) 
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4. 


dg}{sKN,x{sKN)) 


( 11 , 21 ) 


11.4 Numerical Solutiori of Optimal Control 

The numerical methods for solving optimal control problems are generally 
divided into two classes, the indirect methods and the direct methods. 

The indirect methods involve forming the optimality conditions by using 
the calculus of variations and the Pontryagin maximum principle [10, 54]. 
These methods transcribe the optimal control problem into a set of two- 
points boundary value problems in the state and co-state (adjoint) vari- 
ables. The resulting equations are then solved using the shooting methods 
[10, 55]. 

The direct methods involve the transformation of the optimal control prob¬ 
lem into a large scale nonlinear programing problem [11], and then solving the 
resulting constrained nonlinear programming problem using Standard sequen- 
tial quadratic programming (SQP) methods [12, 25]. are also known as dis- 
cretize then optimize methods, because the problem discretization is a prior 
stage to the optimization process [10]. 

The direct methods include 

1. the collocation methods, in which both the control and state variables are 
discretized using the collocation methods. 

2. the control parameterization methods, in which the control variables are 
parameterized and the optimal control problem, becomes an optimal 
parameter selection problem [32]. and, 

3. the iterative dynamic programming (IDP), which is an extension to the 
Bellman dynamical programming concept [35]. 

A classic example of Optimal Control Software based on an indirect 
method is MISER3, [25]. Dieter Kraft (1985) presented the Software package, 
Trajectory Optimization by Mathematical Programming (TOMP), by which 
numerical Solutions of OCPs may be calculated by a direct shooting approach 
[27]. In 2001, J. Kiezenka and L.F.Shampine presented the MATLAB proce- 
dure bvpJf-C as an example of an effective application of indirect collocation 
methods to solve the two-point boundary value problems resulting from the 
Pontryagin maximum principle. In 1997, John T. Betts and W.P Hoffman 
presented the Sparse Optimal Control Software (SOCS) [11]. Another FOR¬ 
TRAN code DIRCOL by O. Von Stryk [1993] is an example of the direct 
collocation approach [57]. 
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11.5 Solving Optimal Control Problems Using Indirect 
Methods 


11.5.1 Numerical Solution Using Indirect Transcription 
Method 


Given the Euler-Lagrange equations: 

^ = 0 

au 

dl-L 

x{t) = — = f{t,x{t),u{t)), x{to) =Xo 

f)nj 

P(i) = --^,p(T)=Pt 
Suppose that Equation 11.22 can be solved such that 

u*{t)=g{x{t),p{t)), 


( 11 . 22 ) 

(11.23) 

(11.24) 


then the following steps can be carried out to find the optimal control and 
optimal trajectory. 

Step one: Divide the interval [to,tf] into N equally spaced sub-intervals, by 
the points sq < si < ■ ■ ■ < sn, where h = Sj+i — Si = 

Step two: Choose an initial guess u = for the control u(t), a maximum 
number of iterations Maxiters and a tolerence e. 

Step three: set n = 0. 

Step four: repeat the following steps: 

Step five: solve the state equations (11.20) with initial condition (11.3) as 
follows: 


Algorithm 1 (Solving state equations) 

5.1 Set x^^^^ = x^. 

5.2 for t = 0,l,---,^-l 

5.2.1 advance to using an integration method of ODEs, such as 

the fourth-order Runge-Kutta method 


ki 

k2 


ks 


3.*+lU) 








Xi^^'> + ^{ki + 2k2 + 2k3 + k4). 
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5.3 interpolate the points = 0, ..., by a cubic spline ip^^^\t) 

5-4 return (sj,) 

Step six: using the state functions x^'^'1, solve the co-state equations as 
follows: 


Algorithm 2 (solving co-state equations) 

fi 1 .pf „Ni^) _ 

0.1 seip - 

6.2 for i = N —1 down to 0 

6.2.1 advance top^ using the an integration method of ODEs, such as 
the fourth-order Runge-Kutta method 


fcl 

= 9{ 


k2 

= 9 

f h ,_.i(n) hki 

fcs 

= 9 1 

f h ,_.i(n) hk2 


= 9{ 

— /i, — hk3,u'^^^^ 

pi(-) 

— — — (fci + 2^2-f 2^3 + fc4, 

6.3 return (sj, 

p ), 

o" 

II 


Step seven: update the control u by 


Step eight: set n = n+l. 

Step nine: repeat the steps five, six, seven and eight until a convergence 
is achieved or the number of maximum iterations is passed, that is — 

< e or n > Maxiters. 

Step ten: in the case of success return 

An Illustrative Example 
Example 11.2 


min J(u) = I{T) + 

U 



[bu^{t)+I{t)] dt 


subject to: 


I{t) = (3{l-I{t))I{t)-au{t)I{t), 1(0) =/o, tG[0,r] 
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'Umin ^ — '^raax 

where the state variable I{t) fulfils the constraints: 

0 < I{t) < 1 

The Hamiltonian of this problem is given by 

= / + bu^ + A(/3(l — /) — (7 + au) */, t g[0,T] 
and the first order necessary conditions are given by 

+ 2bu{t) = 0 ^ u* = - 

ou 2b 

iit) = Pi'^-Iit))Iit)-il + au{t))I{t) 

A(t) = —1 +(2/3/ + 7 + aM(t) —/3)A 


The transversality condition is given by: 


A(T) 


dI{T) 

dI{T) 


Let N he a positive integer and h = ^ is the step size. The interval [0,T] 
is partitioned by N + IS" points to,... ,tN, where U = ih,i = 0,...,N. Let vj « 
u{tj),Zj « I{tj),lj « X{tj),uhj « u{tj + h/2) and zhj « I{tj + h/2) for j = 
We set 


f{zj,Vj) = {j3 — I3zj —"f — avj)zj and g{zj,Vj,lj) = —1.0+(2j3zj+'j+avj — l3)lj. 

Starting from any initial guess for the control, and setting be 

the piecewise cubic Hermite interpolation of u{t) at t = tj + hj2 the classi- 
cal fourth-order Runge-Kutta methods can be used to solve the state and 
co-state equations as follows: 


fcl = fiZj,Vj) 

^2 = f{zj + h*ki/2,uhj) 

ks = f{zj + h*k2/2,uhj) 

ki = fizj + h*k3,Vj+i) 

Zj-^-i = Zj + h/6*{ki + 2k2 + 2k3 + k4),j = 0,...,N—1 

K2 = g{zhj,uhj,lj+i — h/ 2 Ki) 

^3 = g{zhj,uhj,lj^i — h/2K2) 

Ki = g{zj,Vj,lj + i-hK3) 

Ij = Zj+i —/i/6(«:i+2 k 2+ 2^3+ K4), j =1,■ • • ,0 
where zhj is the cubic Hermite interpolation of I{t) at t = tj + /i/2. 
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For T = 20,&= l,a = 0.5,7 = 0.2,/3= 1.0,/o = 0-01, A(r) = 1, iV = 5000 
and £ = 10“^^, the following Python code is used to solve the optimal control 
problem. 


1 import numpy as np 

2 import matplotlib.pylab as plt 

3 

4 def UpdateControl(x, p): 

5 return np.array( [min (u, umax) for u in [max(u, umin) for u ... 

in a*p*x/{2.0*b**2)]]) 

6 def SolveStateEquation(u): 

7 from scipy import interpolate 

8 f = lambda z, v: (bt-bt*z-gm-a*v)*z 

9 th = np.arange(h/2,0, T, h) 

10 uh = interpolate.pchip(t, u)(th) 

11 x[0] = xO 

12 for j in range (len (t)-1) : 

13 kl = f (x [ j] , u [ j ] ) 

14 k2 = f (x [ j ]+h*kl/2, uh [ j ] ) 

15 k3 = f (x [ j ]+h*k2/2, uh [ j ] ) 

16 k4 = f{x[j]+h*k3, u[j + l]) 

17 x[j+l] = x[j] + h/6*{kl+2*k2+2*k3+k4) 

18 return x 

19 

20 def SolveCostateEquation(x, u): 

21 p[N] = pT 

22 th = np.arange(h/2,0, T, h) 

23 uh = interpolate.pchip(t, u)(th) 

24 xh = interpolate.pchip(t, x)(th) 

25 g = lambda z, Im, v: -1.0+(2*bt*z+gm+a*v-bt)*lm 

26 for j in [N-int(i)-l for i in list ( range (len (t)-1))] : 

27 kl = g(x[j + l], p[j + l], u[j + l]) 

28 k2 = g(xh[j], p[j+1]-h*kl/2, uh[j]) 

29 k3 = g(xh[j], p[j+1]-h*k2/2, uh[j]) 

30 k4 = g(x[j], p[j + l]-h*k3, u[j]) 

31 p[j] = p[j+l] “ h/6.0*{kl+2*k2+2*k3+k4) 

32 return p 

33 

34 b, a, gm, bt = 1.0, 0.5, 0.2, 1.0 

35 T = 20.0 

36 xO, pT = 0.01, 1.0 

37 umin, Umax = 0.0, 1.0 

38 N = 10000 

39 t = np.linspace(0.0, T, N+1) 

40 h = T/N 

41 X, p, U = np. ones-like (t) , np. ones-like (t) , 0.5*np. ones-like (t) 

42 

43 X = SolveStateEquation(u) 

44 from scipy import interpolate 

45 p = SolveCostateEquation(x, u) 

46 uold = u 

47 u = UpdateControl(x, p) 

48 Error = np.linalg.norm(u-uold, np.inf) 

49 Iterations = 1 
while Error > le-15: 


50 
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51 uold = u 

52 X = SolveStateEquation(u) 

53 p = SolveCostateEquation(x, u) 

54 u = UpdateControl (x, p) 

55 Error = np.linalg.norm(u-uold, np.inf) 

56 print (Iterations, Error) 

57 Iterations += 1 

58 print (Iterations, ' \t ' , Error) 

59 u = UpdateControl(x, p) 

60 

61 plt. f igure (1) 

62 plt.subplot (1, 2, 1) 

63 plt.plot(t, u, color= ' crimson ' , lw=2) 

64 plt.xlabel (’ Time (t)', fontweight= ' bold ' ) 

65 plt.ylabel (' Optimal Treatment (u(t))', fontweight= ' bold ' ) 

66 plt. grid (True, Is = ' — ') 

67 plt.xticks(np.arange (0, T*l.l, T/10), fontweight= ' bold ' ) 

68 mnu, mxu = np.floor(1000 *min (u))/1000, np.ceil(1010 *max (u))/1000 

69 hu = (mxu-mnu)/10 

70 plt.yticks(np.arange(mnu, mxu+hu, hu), fontweight= ' bold ' ) 

71 plt.axis([0, T, mnu, mxu]) 

72 plt. subplot (1, 2, 2) 

73 plt.plot(t, X, color = 'purple', Iw = 2) 

74 plt.xlabel (' Time (t)', fontweight= ' bold ' ) 

75 plt.ylabel (' Infective Population (I(t))', fontweight= ' bold ' ) 

76 plt. grid (True, Is = ’ — ') 

77 plt.xticks(np.arange (0, T*l.l, T/10), fontweight= ' bold ' ) 

78 mnx, mxx = np.floor( 10*min (x))/10, np.ceil( 10*max (x))/10 

79 hx = (mxx-mnx)/10 

80 plt.yticks (np.arange(mnx, mxx+hx, hx), fontweight= ' bold ' ) 

81 plt.grid(True, Is = '--') 

82 plt.axis([0, T, mnx, mxx]) 

By running the script, we obtain the output: 

runfile(’D:/PyFiles/SolveOCPSI.py’, wdir=’D:/PyFiles’) 
Iteration Error 


1 0 

2 0 

3 0 

4 1 

5 1 

6 9 

7 6 

8 5 

9 3 

10 2 

11 1 

12 1 

13 1 

14 1 


03923191725605521 

0033203169324064197 

0002443004136510607 

7888301100443815e-05 

3080226375083992e-06 

563564312697892e-08 

9922647949471894e-09 

112286149966394e-10 

737765652545022e-ll 

7327029528123603e-12 

9961809982760315e-13 

432187701766452e-14 

1102230246251565e-15 

1102230246251565e-16 
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FIGURE 11.1: The optimal control and optimal trajectory of Example 11.1. 


The optimal treatment and corresponding density of infected population 
are explained figure (11.1) 

Using MATLAB, the optimal treatment and corresponding density of 
infected population can be computed by the following code: 


1 ciear ; clc ; clf ; 

2 global a xO gm b bt N h t T pT ; 

3 a=0.5;b=l; gm = 0.2 ; bt = 1.0; N = 10000 ; 

4 T = 20 ; xO = 0.01 ; pT = 1 ; 

5 h = T/N ; 

6 t = linspace{0, T, N+l) ; 

7 uold = 0.5*ones(l, N+l) ; %p = ones(l/ N+l) ; x = ones(l, N+l) ; 

8 X = SolveStateEquation(uold) ; 

9 p = SolveCostateEquation(x, uold) ; 

10 u = UpdateControl (x, p) ; 

11 

12 Iterations = 1 ; 

13 Error = norm(u-uold, inf) ; 

14 while Error > le-15 

15 uold = u ; 

16 X = SolveStateEquation(u) ; 

17 p = SolveCostateEquation(x, u) ; 

18 u = UpdateControl (x, p) ; 

19 Error = norm(u-uold, inf) ; 

20 disp ( [num2str (Iterations) '\t\t' num2str (Error)]) ; 

21 Iterations = Iterations + 1 ; 

22 end 

23 figure (1) ; 

24 plot{t, u, ' -b ' , 'LineWidth', 2) ; 

25 xlabel ( ' Time (t)') ; 

26 ylabel (’ Control (u(t))') ; 

27 grid on ; 

28 figure (2) ; 
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29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 


plot(t, X, '-r', 'LineWidth', 2) ; 

xlabel ( ' Time (t) ' ) ; 

ylabel ( ’ State {x (t)) ' ) ; 

grid on ; 

function u = UpdateControl(x, p) 
global a b ; 

u = min (max (a*p. *x/(2*b''2) , 0), 1) ; 

end 

function x = SolveStateEquation(u) 
global a xO gm bt N h t T ; 

X = ones(1, N+1) ; 

f = @(z, v) (bt-bt*z-gm-a*v)*z ; 

th = h/2:h:T-h/2 ; 

uh = pchip(t, u, th) ; 

x(l) = xO ; 

for j = 1 : N 

kl = f (x{j) , u(j) ) ; 

k2 = f {x { j)+h/2*kl, uh{j)) ; 

k3 = f{x(j)+h/2*k2, uh{j)) ; 

k4 = f{x(j)+h*k3, u(j + l)) ; 

x(j + l) = x{j) + h/6* {kl + 2*k2 + 2*k3+k4) ; 

end 

end 

function p = SolveCostateEquation(x, u) 
global bt gm a pT N h t T; 
p = ones(1, N+1) ; 

g = @(z, Im, v) -1,0+{2*bt*z+gm+a*v-bt)*lm ; 

th = h/2:h:T-h/2 ; 

uh = pchip(t, u, th) ; 

xh = pchip(t, X, th) ; 

p(N+l) = pT ; 

for j = N : -1 : 1 

kl = g{x(j + l) , p(j + l) , u( j + 1) ) ; 

k2 = g{xh{j), p(j+1)-h/2*kl, uh{j)) ; 

k3 = g{xh{j), p ( j + 1)-h/2*k2, uh{j)) ; 

k4 = g{x(j), p(j + l)-h*k3, u(j)) ; 

P(j) = P(j + 1) - h/6* {kl + 2*k2+2*k3 + k4) ; 

end 

end 


The optimal control and state variables are computed iteratively, until at some 
iteration k, the condition \ < s is fulfilled, where s is a small 

arbitrary constant (selected to be 10“^^ in the example). In this example, it 
is noticed that fourteen iterations were needed to compute the optimal control 
and optimal trajectory. 


Example 11.3 In this example we solve an optimal control problem found 
in [56]. It describes Glucose-Insulin interaction. The optimal control problem 
is formnlated as follows: 


min lim 


{xi{t) — Xd)^ pu^{t)dt 
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subject to: 


xi{t) = —mixi{t)— m2X2{t), xi{ 0 ) = xio 
X 2 {t) = -m3X2{t)+ u{t), X2{0) = X 20 

where u(t) accounts for the rate of infusion of exogenous hormone. 

Setting Xdl00,p = 10,mi = 0.0009,m 2 = 0.0031,ms = 0.0415 and starting 
from a;i(0) = 2 : 10 , 3:20 = X 20 , the following Python code solves the optimal 
control problem. 


1 import numpy as np 

2 import matplotlib.pylab as plt 

3 

4 def UpdateControl(p): 

5 return np.array( [max (u, 0) for u in -p[:, l]/(2.0*r)]) 

6 

7 def SolveStateEquation(u): 

8 from scipy import interpolate 

9 f = lambda z, v: np,array([-ml*z[0]-m2*z[1], -m3*z[l]+v]) 

10 th = np.arange(h/2,0, T, h) 

11 uh = interpolate .pchip (t, u) (th) 

12 X = np. zeros ( ( len (t) / 2), 'float') 

13 xlO, x20 = 300.0, 0.0 

14 x[0, :] = np.array([xlO, x20]) 

15 for j in range (len (t)-1) : 

16 kl = f (x [ j] , u [ j ] ) 

17 k2 = f (x[ j]+h*kl/2, uh[j]) 

18 k3 = f (x [ j ]+h*k2/2, uh [ j ] ) 

19 k4 = f(x[j]+h*k3, u[j + l]) 

20 x[j+l] = x[j] + h/6*(kl+2*k2+2*k3+k4) 

21 return x 

22 

23 def SolveCostateEquation(x): 

24 p = np.zeros (( len (t), 2), 'float') 

25 th = np.arange(h/2,0, T, h) 

26 xh = interpolate.pchip(t, x)(th) 

27 g = lambda z, Im: np.array([ml*lm[0]-2*z[0]+2*xd, ... 

m2*lm[0]+m3*lm[1]]) 

28 for j in [N-int(i)-l for i in list ( range (len (t)-1))] : 

29 kl = g(x[ j + 1] , p[ j + 1] ) 

30 k2 = g(xh[j], p [ j + 1 ]-h*kl/2) 

31 k3 = g(xh[j], p [ j + 1 ]-h*k2/2) 

32 k4 = g(x[j], p[j + l]-h*k3) 

33 p[j] = p[j+l] “ h/6.0*(kl+2*k2+2*k3+k4) 

34 return p 

35 

36 xd, r, ml, m2, m3 = 100.0, 10.0, 9.0e-4, 3.1e-3, 4.15e-2 

37 T = 90.0 

38 umin, Umax = 0.0, 1.0 

39 N = 30000 

40 t = np.linspace(0.0, T, N+1) 

41 h = T/N 

42 u = np . ones-like (t) 


43 
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44 X = SolveStateEquation (u) 

45 from scipy import interpolate 

46 p = SolveCostateEquation(x) 

47 uold = u 

48 u = UpdateControl(p) 

49 Error = np.linalg.norm(u-uold, np.inf) 

50 print ( ' Iteration\t\t Error\n') 

51 Iterations = 1 

52 

53 while Error > le-15: 

54 uold = u 

55 X = SolveStateEquation(u) 

56 p = SolveCostateEquation(x) 

57 u = UpdateControl (p) 

58 Error = np.linalg.norm(u-uold, np.inf) 

59 print (Iterations, '\t\t'. Error) 

60 Iterations += 1 

61 

62 plt. f igure (1) 

63 plt.plot(t, u, color= ' crimson ’ , lw=2) 

64 plt.xlabel (’ Time (t)', fontweight= ' bold ' ) 

65 plt.ylabel (' Optimal Treatment (u(t))', fontweight= ' bold ' ) 

66 plt.grid(True, Is = '--') 

67 plt.xticks(np.arange(0, T*l.l, T/10), fontweight= ' bold ' ) 

68 mnu, mxu = np.floor(0.1 *min (u))*10, np.ceil(0. l*max (u))*10 

69 hu = (mxu-mnu)/10 

70 plt.yticks(np.arange(mnu, mxu+hu, hu), fontweight= ' bold ' ) 

71 plt.axis([0, T, mnu, mxu]) 

72 plt. savef ig ( ' OCPGIl . eps ' ) 

73 plt.savefig( 'OCPGIl .png' ) 

74 

75 plt. f igure (2) 

76 

77 plt.subplot(1, 2, 1) 

78 plt.plot(t, x[:, 0], color = 'purple', Iw = 2, label = 'Glucose') 

79 plt.xlabel (' Time (t)’, fontweight= ’ bold ' ) 

80 plt.ylabel (' Glucose (xl(t) mg/dl)', fontweight= ' bold ' ) 

81 plt. grid (True, Is = ' -- ' ) 

82 plt.xticks(np.arange(0, T*l.l, T/10), fontweight= ' bold ' ) 

83 mnx, mxx = np.floor(0.1 *min (x[:, 0]))*10, np.ceil(0. l*max (x[:, 

0]))*10 

84 hx = (mxx-mnx)/10 

85 plt.yticks(np.arange(mnx, mxx+hx, hx), fontweight= ' bold ' ) 

86 plt.grid(True, Is = '--') 

87 plt.axis([0, T, mnx, mxx]) 

88 

89 plt.subplot (1, 2, 2) 

90 plt.plot(t, x[:, 1], color = 'orangered', Iw = 2, label = ... 

' Insulin ' ) 

91 plt.xlabel (’ Time (t)', fontweight= ' bold ' ) 

92 plt.ylabel (' Insulin (x2(t) mg/dl)', fontweight= ' bold ' ) 

93 plt.grid(True, Is = ’--') 

94 plt.xticks(np.arange(0, T*l.l, T/10), fontweight= ' bold ' ) 

95 mnx, mxx = np.floor(0.1 *min (x[:, 1]))*10, np.ceil(0. l*max (x[:, 

1]))*10 

96 hx = (mxx-mnx)/10 

97 plt.yticks(np.arange(mnx, mxx+hx, hx), fontweight= ' bold ' ) 
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98 plt. grid (True, Is = ' — ') 

99 plt.axis([0, T, mnx, mxx] ) 

100 plt. savef ig ( ' OCPGI2 . eps ' ) 

101 plt. savefig ( 'OCPGI2 .png' ) 

The optimal infusion rate of Insulin and the corresponding interaction of 
Glucose and Insulin are explained in Figures 11.2a-11.2b. 



(a) optimal rate of infusion of exogeneous Insulin 



(b) Glucose and Insulin dynamics, left figure: glucose, right figure: insulin. 


FIGURE 11.2: Solution of the optimal control problem of Glucose-Insulin 
interaction. 


The MATLAB code for solving the optimal control problem of the glucose- 
insulin interaction is as follows: 


1 ciear ; clc ; clf ; 

2 global ml m2 m3 xlO x20 N h t T b xd ; 

3 xd = 100.0; b = 10.0; ml = 9.0e-4; m2 = 3.1e-3; m3 = 4.15e-2 ; 

4 T = 90.0; N = 30000 ; 

5 xlO = 300.0 ; x20 = 0.0 ; 
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27 

28 

29 
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31 
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33 

34 
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h = T/N ; 

t = linspace{0, T, N+1) ; 

uold = 0.5+ones{N+1, 1) ; %p = ones(l, N+1) ; x = ones(l, N+1) ; 

X = SolveStateEquationGI(uold) ; 
p = SolveCostateEquationGI(x) ; 

u = UpdateControlGI(p) ; 


Iterations = 1 ; 

Error = norm (u-uold, inf) ; 
while Error > le-15 
uold = u ; 

X = SolveStateEquationGI(u) ; 

p = SolveCostateEquationGI(x) ; 

u = UpdateControlGI(p) ; 

Error = norm(u-uold, inf) ; 

disp ( [num2str (Iterations) \t\t num2str (Error)]) ; 

Iterations = Iterations + 1 ; 

end 


figure (1) ; 

plot{t, u, '“b', 'LineWidth', 2) ; 

xlabel ( ' Time (t)') ; 

ylabel (’ Optimal Infusion Rate (u(t))') ; 

grid on ; 


figure (2) ; 

subplot (1, 2, 1) ; 

plot{t, x(:, 1), '-r', 'LineWidth', 2) ; 

xlabel (' Time (t)') ; 

ylabel ( 'Glucose (x_l(t)) ' ) ; 

grid on ; 

subplot (1, 2, 2) ; 

plot{t, x(;, 2), '-r', 'LineWidth', 2) ; 

xlabel (' Time (t)') ; 

ylabel ( ' Insulin (x_2(t)) ' ) ; 

grid on ; 


function u = UpdateControlGI(p) 
global b ; 

u = max(-p{:, 2)/(2*b), 0) ; 

end 


function x = SolveStateEquationGI(u) 
global ml m2 m3 xlO x20 N h t T ; 

X = ones(N+l, 2) ; 

f = @(z, v) [-ml*z(1)-m2*z(2), -m3*z{2)+v] ; 

th = h/2:h:T-h/2 ; 

uh = pchip(t, u, th) ; 

x(l, :) = [xlO, x20] ; 

for j = 1 : N 

kl = f {x{j, :) , u{j) ) ; 

k2 = f{x{j, :)+h*kl/2, uh(j)) ; 

k3 = f{x{j, :)+h*k2/2, uh(j)) ; 

k4 = f{x{j, :)+h*k3, u{j+l)) ; 

x(j+l, :) = x(j, :) + h/6*(kl+2*k2+2*k3+k4) ; 


62 


end 
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63 end 

64 

65 function p = SolveCostateEquationGI(x) 

66 global xd ml m2 m3 N h t T; 

67 p = ones(N+l, 2) ; 

68 g = @(Z/ liT^) (1)-2*z {1)+2 *xd, m2*lm (1)+m3*lm (2) ] ; 

69 th = h/2:h:T-h/2 ; 

70 xh = pchip(t(:), x’, th ( : ) ) ’ ; 

71 p(N+l, :) = zeros(l, 2) ; 

72 for j = N : -1 : 1 

73 kl = g{x(j + l, :), p{j + l, :)) ; 

74 k2 = g{xh{j, :), p{j + l, :)-h*kl/2) ; 

75 k3 = g{xh{j, :) , p{j + l, :)-h*k2/2) ; 

76 k4 = g(x(j, :), p{j + l, :)-h + k3) ; 

77 P(j/ O = p(j + l/ •) ~ h/6.0* (kl+2*k2 + 2 *k3+k4) ; 

78 end 

79 end 


11.6 Solving Optimal Control Problems Using Direct 
Methods 

Direct transcription methods are well-known class for solving optimal con¬ 
trol problems. Direct transciption methods are also known as discretize then 
optimize methods, because the problem discretization is a prior stage to the 
optimization process [10]. Methods of discretization such as the Runge-Kutta 
methods [50], splines [24], collocation methods [57], etc., are used for the dis¬ 
cretization of the state equations, whereas some numerical quadrature method 
such as the trapezoidal rule or the Simpson’s rule is used for the evaluation 
of the objective function. 

One of the efficient direct transciption methods for solving the optimal 
control problems, is the control parameterization technique [58, 59, 60]. In a 
CPT, the time domain is partitioned by a number of switching points at which 
the control variables are evaluated, then each switching interval is partitioned 
by a number of quadrature points at which the state variables are evaluated. 
At each switching interval, the control variable is approximated by a constant 
or linear piece-wise continuous function [58]. 

At the end of the discretization stage, the optimal control problem is trans- 
formed into a large or medium scale finite dimensional nonlinear programming 
problem (NLP) [11]. The resulting NLP can be solved by using any nonlinear 
programming Software, such as the MATLAB’s optimization toolbox [17], the 
SQP [12], the FSQP [29], etc. 
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11.6.1 Statement of the Problem 

In this section we consider an optimal control problem of the form: 


rf 

miiiip{x{tf)) + / Lo{x{t),u{t))dt 
«eu 

(11.25) 

subject to the dynamics: 


II 

(11.26) 

with initial conditions 

£c(fo) = 

(11.27) 

where a;,(^:R—^-R"", m:R—>■ R”^ and G R". 

The System is also subject to state inequality constraints: 


I{x{t)) < 0 

(11.28) 

and state equality constraints 


E{x{t))=0 

(11.29) 


where J : R” —>■ and E : —> R® are differentiable with respect to x. 

In the two following sections we describe the solution of optimal control 
problems using control parameterization technique (CPT) and imple- 
ment it in MATLAB. Also, we will discuss the solution of optimal control 
problems using the Python’s Gekko package. 

11.6.2 The Control Parameterization Techniqne 

For the discretization of the control variables u{t), let Nc be a positive integer 
and hc = ° then so(= to) < si < ... < sn^ = tf where Si+i = Si + h is a 

partition for the interval [to, t/ ] • Let us assume that, in an interval [sj, sj +1 ] the 
value of the control function is a constant {i.e.Ui{t) = Vj) 

or a linear function {i.e. Ui{t) = + The constants v* or a\ and 6^ 

are called the control parameters. An interval is called a switching 

interval. Hence, there are Nc swithching intervals [sj, Sj-^.l\,j = 0,1,..., Nc — 
1. For simple discussion, we assume that in Uj{t) = Vj where Vj is a 

constant. 

For the discretization of the state variables, each switching interval 
[si,Si+i) is divided into Q sub-intervals, where Q is a fixed positive inte¬ 
ger. The time interval [to,tf] is divided into N = NcQ subintervals. Let 
h = {tf — to)/N and U = to + ih. At a time U suppose that the state vari- 
able Xj{ti) is approximated by the value cc*; j = 1,..., = 0,..., A^. We also 

notice that 

tp to f/ ^0 hc 
N ^ NcQ 
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A point ti,i = 0,...,N lies in a switching interval where k = 

[i/hc\- Hence, 

Uj{ti) = Vj and XmiU) = 

Now, the control Uj{t),t G [to,tf] is given by 

The Mayer part of the objective function ip(^x{tf)) is evaluated at the time 
In- i-e. (f(^x{tf)) = (p(^x{tN))- We use the Simpson’s quadrature to approx¬ 
imate the Bolza part of the objective function, provided that N is an even 
integer. Let 


Lq = Lo{x{ti),u{ti)) = Lo{x{ti),v^), k = li/hc\, 


then, 



Lo(^t,x{t),u{t)) dt 


2=1 


22-1 

0 


7-22 

2=1 


The objective function is approximated by 


L^J L^J 

J{u{t)) « g,{x^) + J (lO + L^) + J J (^^ 30 ) 

i=l i=l 

The equality constraints (11.29) become: 

£;(x*) = 0, 0<i<A^ (11.31) 

The inequality constraints (11.28)become, 

/(®*)<0, 0<z<fV (11.32) 

The classical fourth-order Runge-Kutta method is used for the discretiza- 
tion of the state equations as follows. 


fei = f{x\u^) 

k2 = /(a:*-l-ft.fci/2,M*+2) 

fes = /(a:*-l-ft.fc2/2,M*+2) 

^4 = /(a;* + 

= a;*-I-— (fci-I-2fc2 + 2fc3-t-fe4) (11.33) 

6 

where = u{ti + h/2) can be approximated from some interpolation 

method, such the piecewise cubic Hermite interpolating polynomials. 
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The continuous optimal control problem described by the equations 
(11.25), (11.26), (11.27), (11.28) and (11.29) is transformed into the discrete 
nonlinear programming problem: 


minimize (p(x^)-l —(Lq-|-L^ 
u£R™(l + iVc) 3 

i=l i=l 

(11.34) 

subject to: 

x^~^^ — a;* — ^ (fci -1- 2^2 -1- 2^3 -|- /24) = 0 

(11.35) 

subject to the initial condition 



a:(0) 

-x° = 0 

(11.36) 

subject to the equality constraints: 



E(x^)= 0 , 

z = 0 ,l, 2 ,...,N 

(11.37) 

the inequality constraints: 



l(x^)< 0 , 


(11.38) 


Finally both the control variables mj, , fc = 1,2,..., m and the state variables 
Xi,i= 1,2,...,n are mapped into one vector Y. xj is mapped into Y{i — 1 + 
(i — l)iV + j),i = 1,2,... ,n;j = 0,1,... ,1V, and is mapped into Y'((fc — 1) + 
n+ (n — 1)7V+ (/c — l)A:"c+ L(^~ l)/OJ))^ = 1,2,... ,to;^ = 0,1,... ,iV. The total 

length of the vector Y is n(l + A^) + m(l + Nc). 

The problem described by (11.34), with the constraints (11.35) or (11.35)- 
(11.37) then becomes: 


minimize 

Y ^■^n{\ + N) + Tn[l + Nc) 

subject to the equality and inequality constraints: 

E{Y) = 0, 

I{Y) < 0 

and subject to the initial conditions: 

y(j — 1-I-(i — l)iV) — = 0, t = 1,2,... ,n 


11.6.2.1 Examples 

Example 11.4 This example is taken from [26]. 

minimize J(u) = S(T) +I(T)-R{T)+ [ (S{t)+I(t)-R(T) + Ciu'^it) + C2v'^(t))dt 

u(t) 


0 
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subject to: 

S{t) = K-aS{t)I{t)-{pL + u)S{t),S{{)) = So,t&[Q,T] 
i{t) = aS{t)I{t)-{pi + p)I{t)-v{t)I{t),m = h,t(i[Q,T] 

R(t) = l3I{t) + u{t)S{t)-pLR{t) + v{t)I{t),R{Q) = Ro,t£[Q,T] 

where 

0 < S{t),I{t),R{t) < 1 

and 

'^min ^ — '^max and Vmin ^ — '^m.ax 

To solve this problem with MATLAB, the modebs paremeters Ci, (72, A,a,/i 
and (3 shall be declared as global variables. 

Based on the Simpson’s rule, a function SIRObjective is implemented for 
the evaluation of the objective function. 


1 function OptimalCost = SIRObjective(x) 

2 global C1 C2 N NC h Q ; 

3 S = X (1:1+N) ; 

4 I = x(2+N:2+2*N) ; 

6 R = X(3+2*N:3+3*N) ; 

6 u = X(4+3*N:4+3*N+NC) ; 

7 V = x(5+3*N+NC: 5+3*N+2*NC) ; 

8 L = @ (j) ... 

S(j)+I (j)-R(j)+Cl*u(l + floor(j/Q) ) "2 + C2*v{l + floor(j/Q) ) "2 ; 

9 OptimalCost = S(end) + I(end)-R(end) + h/3*(L(1)+L(end)) ; 

10 for j = 2 : N 

11 if mod(j, 2) == 0 

12 OptimalCost = OptimalCost + 4*h/3*(L(j)) ; 

13 else 

14 OptimalCost = OptimalCost + 2*h/3*(L(j)) ; 

15 end 

16 end 


Write a function SIRConstsRK4 to return the equality and inequality con- 
straints in two vectors ic and ec. The fourth order Runge-Kutta method is 
used for the discretization of the state equations, where the piecewise cubic 
Hermite interpolation polynomials are used to interpolate the state and con¬ 
trol variables at the points tj + h/2;j = 1,...,7V. The MATLAB code of the 
functionSIRConstsRK4 is as follows. 


1 function [ic, ec] = SIRConstsRK4(x) 

2 global N NC Q h beta mu alpha Lambda t ; 

3 S = X (1;1+N) ; 

4 I = X (2+N:2 + 2*N) ; 

5 R = X (3+2*N:3 + 3*N) ; 

6 u = X (4 + 3*N:4 + 3*N+NC) ; 
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7 
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9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
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22 

23 
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25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 


36 

37 

38 

39 

40 


41 


42 


43 

44 

45 

46 


47 


48 


49 


50 

51 

52 

53 

54 

55 


V = x(5+3*N+NC: 5+3*N+2*NC) ; 


th = t(l:N)+h/2.0 ; 


Sh = pchip(t, 
Ih = pchip(t, 
Rh = pchip(t, 
uh = pchip(t, 
vh = pchip(t, 
umin = 0.05 ; 
Umax = 0.60 ; 
vmin =0.1 ; 
vmax =0.9 ; 
SO = 0.9 ; 

10 = 0,07 ; 

RO = 0.03 ; 


S, th) ; 

I, th) ; 

I, th) ; 

u(l+floor ((0:N)/Q)), 
v(l+floor ((0:N)/Q)), 


ic = zeros (4 + 4*NC, 1) ; 
ec = zeros (3*N+5, 1) ; 
ic(l:l+NC) = umin - u ; 
ic(2+NC:2+2*NC) = u - umax ; 


th) ; 
th) ; 


ic(3+2*NC:3+3*NC) = vmin-v ; 
ic(4+3*NC:4+4*NC) = v - vmax ; 

fl = @(x, y, z, v) Lambda-alpha*x*y-(mu+v)*x ; 

f2 = @ (x, y, z, v) alpha*x*y- (mu+beta) *y-v*y ; 

f3 = @ (x, y, z, u, v) beta*y+u*x-mu*z+v*y ; 

for j = 1 : N 

kll = fl(S(j), I(j), R{j), u(l + floor(j/Q) ) ) ; 

kl2 = fl(S(j)+h/2*kll, Ih(j), Rh(j), uh(j)) ; 
kl3 = fl(S(j)+h/2*kl2, Ih(j), Rh(j), uh(j)) ; 
kl4 = fl(S(j)+h*kl3, I{j+1), R(j+1), ... 

u(l + floor{ (j + l)/Q) ) ) ; 


k21 = f2(S(j), I(j), R{j), v(l + floor (j/Q) ) ) ; 

k22 = f2(Sh(j), I ( j)+h/2*k21, Rh ( j ) , vh ( j ) ) ; 
k23 = f2(Sh(j), I(j)+h/2*k22, Rh(j), vh(j)) ; 
k24 = f2(S(j + l), I(j)+h*k23, R(j + 1), ... 

v(l+floor {(j+l)/Q))) ; 


k31 = f3(S(j), I(j), R{j), u(l + floor(j/Q) ) , 
V (1+floor (j/Q))) ; 


k32 = f3(Sh(j), Ih(j), R(j)+h/2*k31, uh ( j) , vh(j)) 
k33 = f3(Sh(j), Ih(j), R(j)+h/2*k32, uh(j), vh(j)) 
k34 = f3(S(j + l), I(j + 1), R(j)+h*k33, ... 

u(l + floor{ (j + l)/Q) ) , v(l + floor ( (j + l)/Q) ) ) ; 

ec(j) = S(j + 1) - S{j) - h/6 * ... 

(kll+2*kl2+2*kl3+kl4) ; 
ec(N+j) = I(j + 1) - I{j) - h/6 * ... 

(k21+2*k22+2*k23+k24) ; 
ec(2*N+j) = R(j + 1) - R{j) - h/6 * ... 
(k31+2*k32+2*k33+k34) ; 


end 


ec(3*N+l) = S(1)-S0 ; 
ec(3*N+2) = I(1)-10 ; 
ec (3*N+3) = R(l) -RO ; 
ec(3*N+4) = u(end)-u (end-1) ; 

ec(3*N+5) = V (end) -V (end-1) ; 





312 


Solving Optimal Control Problems 


Now, we write a MATLAB script SolveSIROCP.m which uses the functions 
SIRObjective and SIRConsts to find the solution of the optimal control prob- 
lem. 


1 ciear ; clc ; 

2 global N NC Q h Lambda beta mu alpha C1 C2 t ; 

3 Lambda = 0.05 ; mu = 0,05 ; alpha = 2 ; beta = 0.6 ; 

4 10 = 0 ; 

5 T = 5 ; 

6 C1 = 2 ; 

7 C2 = 0.5 ; 

8 h = 0.025 ; 

9 N = T/h ; 

10 Q = 2 ; 

11 NC = N/Q ; 

12 t = linspace{t0, T, 1+N) ; 

13 tc = linspace{t0, T, 1+NC) ; 

14 SO = 0.9 ; 

15 10 = 0.07; 

16 RO = 0.03 ; 

17 xO = zeros (3*N+2*NC + 5, 1) ; 

18 

19 Options = optimset( 'LargeScale' , 'off', 'Algorithm', ... 

'active-set' , 'Display', 'Iter', 'Maxiter', inf, ... 

'MaxFunEvals' , inf, 'TolFun', le-6, 'TolX', le-6, ... 

' TolFun' , le-6) ; 

20 [x, fval] = fmincon(@SIRObjective, xO, [],[],[],[],[],[], 

@SIRConstsRK4, Options) ; 

21 

22 S = X (1 :1+N) ; 

23 I = x(2+N:2+2*N) ; 

24 R = x(3+2*N;3+3*N) ; 

25 u = x(4+3*N:4+3*N+NC) ; 

26 V = X(5+3*N+NC:5+3*N+2*NC) ; 

27 tc = 0 : hc : T ; 

28 us = pchip (tc, u, t) ; 

29 vs = pchip (tc, V, t) ; 

30 

31 f igure (1) ; 

32 

33 subplot(l, 2, 1) ; 

34 plot(t, us, '-b' , ' LineWidth ' , 2) ; 

35 xlabel ( 'Time (t)') ; 

36 ylabel ( ’Vaccination Strategy (u(t))') ; 

37 grid on ; 

38 set (gea, 'XTick', 0:0.5:T) ; 

39 axis {[0, T, 0, 1]) ; 

40 

41 subplot(l, 2, 2) ; 

42 plot(t, vs, '-r', 'LineWidth', 2) ; 

43 axis{[0, T, -0.1, 1]) ; 

44 xlabel ( ' Time (t) ' ) ; 

45 ylabel ('Treatment (v(t))') ; 

46 grid on ; 

set (gea, 'XTick', 0:0.5;T) ; 


47 
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48 axis([0, T, 0, 0.2]) ; 

49 

50 f igure (2) ; 

51 

52 subplot(l, 3, 1) ; 

53 plot{t, S, 'b\ 'LineWidth', 2) ; 

54 xlabel ( ' t' ) ; 

55 ylabel ( 'Susceptibles (S (t)) ' ) ; 

56 grid on ; 

57 set {gea, 'XTick', 0:0.5;T) ; 

58 axis([0, T, 0.0, 1]) ; 

59 

60 subplot(l, 3, 2) ; 

61 plot(t, I, 'r', ' LineWidth 2) ; 

62 xlabel ( ' Time (t) ' ) ; 

63 ylabel (’Infected Population (I(t))') ; 

64 grid on ; 

65 set (gea, 'XTiek', 0:0.5;T) ; 

66 axis([0, T, 0.0, 0.2]) ; 

67 

68 subplot(l, 3, 3) ; 

69 plot(t, R, 'm', 'LineWidth', 2) ; 

70 xlabel ( ' Time (t) ' ) ; 

71 ylabel ('Reeovered Population (R(t))') ; 

72 grid on ; 

73 set (gea, 'XTiek', 0:0.5:T) ; 

74 axis{[0, T, 0.0, 0.8]) ; 

The optimal vaccination and treatment strategies and corresponding ratios 
of suceptible, infected and reeovered populatione are shown in Figures 11.Sa¬ 
li.3b. 

11.6.3 The Gekko Python Solver 

The gekko package can be used to solve dynamical optimization problem at 
mode 6 (Nonlinear control / dynamic optimization (CTL)). The objec- 
tive function shall be written in Mayer’s form, by adding a new variabe X 
whose derivative is the integrand part of the objective function, and the prob¬ 
lem will be to minimize a function ip(X(tf)). The following example shows 
how gekko can be used to solve an optimal control problem. 

Example 11.5 We consider the optimal control problem of the SIR model 
described in Example 11.4, and use the Gekko Python package to solve it. 

The Python code SolveGekSIR uses the Gekko Python to solve the prob¬ 
lem of example 11.4. 


1 from gekko import GEKKO 

2 import numpy as np 

3 import matplotlib.pyplot as plt 

4 

5 Lambda = 0.05 ; mu = 0,05 ; alpha = 2 ; beta = 0.6 

6 tO = 0 
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Time (t) Time (t) 

(a) optimal vaccination and treatment strategies 





(b) Susceptible, infected and recovered populations 

FIGURE 11.3: Solution of the optimal control problem with dynamics gov- 
erned by an SIR model, using the MATLAB optimization toolbox. 


7 T = 5 

8 C1 = 2 

9 C2 = 0.5 

10 N = 200 #nuiiiber of subintervals [t-j, t-{j + l}] 

11 m = GEKKO(remote=False) #no remote server is required 

12 m.time = np.linspace (t0, T, N+1) #discretizating the interval ... 

[tO, T] by N+1 points 

13 S, I, R = m.Var{value=0,9), m.Var(value=0.07), ... 

m.Var(value=0.03) #Initializing the state variables 

14 u, V = m.Var{lb=0.0, ub=0.8), m.Var(lb=0.05, ub=1.0) ... 

#Initializing the control variables 

15 X = m.Var(value=0.0) #The bolza part of the objective function 

16 p = np . zeros (N+1) 

17 p[N] = 1.0 

18 final = m,Param (value=p) 

19 # Equations 

20 m.Equation(S.dt() == Lambda-alpha*S*I-(mu+u)*S) #First state ... 

equation in S 

21 m.Equation(I.dt() == alpha*S*I-(mu+beta)*I-v*I) #Second state ... 

equation in I 

22 m.Equation(R.dt() == beta*I+u*S-mu*R+v*I) #Third state ... 

equation in R 
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23 m.Equation(X.dt() == S+I-R+Cl*u**2+C2*v**2) #Integrand part of ... 

the objective function 

24 m.Obj ((X+S+I-R) * final) #The objective function 

25 m.options.IMODE = 6 # Mode 6 is the optimal control mode 

26 m. solve () 

27 t = m.time 

28 

29 plt. f igure (1) 

30 plt. subplot (1, 2, 1) 

31 plt.plot(t[1:], u[l:], color= 'royalblue' , Iw = 2) 

32 plt . xlabel (' Time (t)', fontweight= ' bold ' ) 

33 plt . ylabel ( ' Vaccination (u(t))', fontweight= ' bold ' ) 

34 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

35 mxu = np.ceil (10*max (u) ) /10 

36 plt . yticks(np . arange (0, mxu+mxu/10 , mxu/10 ), fontweight= ' bold ' ) 

37 plt.grid(True, ls='--') 

38 plt.axis([0,0, T, 0.0, np.ceil{ 10*max (u))/10]) 

39 

40 plt.subplot (1, 2, 2) 

41 plt.plot (t [ 1:], v[l:], color=’red', Iw = 2) 

42 plt.xlabel ('Time (t)', fontweight= 'bold' ) 

43 plt.ylabel ('Treatment (v(t))', fontweight= ’bold' ) 

44 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

45 mxv = np.ceil (10*max (v))/10 

46 plt.yticks(np.arange(0, mxv+mxv/10, mxv/10), fontweight= 'bold' ) 

47 plt.grid(True, ls='—') 

48 plt.axis([0,0, T, 0.0, np.ceil{ 10*max (v))/10]) 

49 

50 plt . f igure (2 ) 

51 plt . subplot ( 1 , 3, 1) 

52 plt.plot(t, S, color= ’ darkblue ' , Iw = 2) 

53 plt.xlabel (’ Time (t)’, fontweight= ’ bold ' ) 

54 plt.ylabel ('Suceptible Population (S(t))', fontweight= 'bold’ ) 

55 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

56 mxS = np.ceil (10*max (S)) /10 

57 plt.yticks(np.arange(0, mxS+mxS/10, mxS/10), fontweight= 'bold' ) 

58 plt.grid (True, ls='--') 

59 plt.axis([0,0, T, 0.0, np.ceil{ 10*max (S))/10]) 

60 plt.savefig( 'SIRConts.eps' , dpi=1200) 

61 plt.savefig( 'SIRConts.png' , dpi=12 00) 

62 

63 plt . subplot ( 1 , 3, 2) 

64 plt.plot(t, I, color= ' crimson ' , Iw = 2) 

65 plt . xlabel (' Time (t)', fontweight= ' bold ' ) 

66 plt.ylabel ('Infected Population (I(t))', fontweight= 'bold' ) 

67 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

68 mxl = np.ceil( 10*max (I))/10 

69 plt . yticks(np . arange(0, mxl+mxl/10 , mxl/10 ), fontweight= ’ bold ' ) 

70 plt . grid(True, ls=' —') 

71 plt.axis([0,0, T, 0.0, np.ceil{ 10*max (I))/10]) 

72 

73 plt. subplot (1, 3, 3) 

74 plt.plot(t, R, color= 'darkgreen' , Iw = 2) 

75 plt . xlabel (' Time (t)’, fontweight= ' bold ' ) 

76 plt.ylabel ('Recoved Population (R(t))', fontweight= 'bold' ) 

77 plt.xticks (np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

78 mxR = np . ceil (10*max (R) ) /10 
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79 plt.yticks(np.arange(0, mxR+mxR/10, mxR/10), fontweight= 'bold' ) 

80 plt.grid(True, ls='--') 

81 plt. axis ([ 0,0, T, 0.0, np . ceil { 10*inax (R) )/10 ] ) 

82 plt. savefig ( ' SIRStates . eps ' , dpi=1200) 

83 plt.savefig ('SIRStates.png' , dpi=1200) 


In Figure 11.4 the optimal vaccination and treatment strategies and the 
corresponding snsceptible, infected and recovered populations, obtained by 
the Gekko Python package are shown. 

Example 11.6 In this example we consider the pest control problem found 
in [51]. The purpose is to find the optimal spraying schednle to eradicate the 
number of insect preys of spiders in agroecosystems. The optimal control is 
given by: 


minimize J{u) 

u{t) 








(a) optimal vaccination and treatment strategies 



Tiined) Tiined) Tiine(t) 

(b) Snsceptible, infected and recovered populations 


FIGURE 11.4: Gekko solution of the optimal control model with dynamics 
governed by an SIR model. 



























































Solving Optimal Control Problems Using Direct Methods 
subject to: 


317 


x{t) = -cx{t)y{t)-a{l-q)u{t),x{0) = Xo,tG[0,T] 

y{t) = y{t){-a + kbz{t) + kcx{t))-aKqu{t),y{Q) = yo,t&[Q,T] 

z{t) = ez{t)(^-^^^^-by{t)z{t)-aqu{t),z{Q) = ZQ,t&[Q,T] 


where 

^min — ^ '^max 

The values of model parameters are as follows: r = l,e = 2.5,a = 3.1,6 = 
1.2,c=0.2,a = 0.7,g = 0.9,fc= 1.0,y = 1000,lT = 5,7s: = 0.01,r = 50 and ^ = 
0.05,0.1. 

The Python code is: 


1 # ! /usr/bin/env pythonS 

2 from gekko import GEKKO 

3 import numpy as np 

4 import matplotlib.pYplot as plt 

5 

6 r=1.0;e =2.5;a=3.1;b=l,2;c=0.2;al=0.7; 

7 q= 0.9 ; k= 1.0 ; V= 1000; W= 5 ; K= 0.01 ; D = 0.5 ; 

8 T = 50 ; 

9 tO = 0 

10 N = 300 

11 m = GEKKO(remote=False) 

12 m.time = np.linspace(t0, T, N+1) 

13 X, Y, z = m.Var(value=3.1), m.Var(value=3.7), m.Var(value=2.2) 

14 u = m.Var(lb=0.00, ub=l,0) 

15 X = m. Var (value=0.0) 

16 p = np.zeros(N+1) # mark final time point 

17 p [N] = 1.0 

18 final = m.Param (value=p) 

19 # Equations 

20 m.Equation(x.dt() == r*x*(1-x/W)-c*x*y-al*(1-q)*u) 

21 m.Equation(y.dt() == y*(-a+k*b*z+k*c*x)-al*K*q*u) 

22 m.Equation(z.dt() == e*z*(1-z/V)-b*y*z-al*q*u) 

23 m.Equation(X.dt{) == z+D/2*u**2) 

24 m.Obj (X * final) # Objective function 

25 m.options.IMODE = 6 # optimal control mode 

26 m. solve () 

27 t = m.time 

28 

29 plt.figure(1, figsize=(16, 16)) 

30 plt.subplot(2, 2, 1) 

31 plt.plot(t[1:], u[l:], color= ’purple' , Iw = 2) 

32 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

33 plt.ylabel (' Optimal Spray Schedule (u(t))', fontweight= 'bold' ) 

34 plt.xticks(np.arange (0, T+T/10, T/10), fontweight= 'bold' ) 

35 mnu = np.floor (1000*min (u))/1000 
mxu = np.ceil (1000*max (u))/1000 


36 
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(a) Solution of the pest optimal spraying schedule for ^ = 0.05 






(b) Solution of the pest optimal spraying schedule for ^ = 0.1 

FIGURE 11.5: Gekko solution of the pest control optimal control problem. 


37 plt.yticks(np.arange(mnu, mxu+mxu/10000, (mxu-mnu)/10), ... 

fontweight= 'bold' ) 

38 plt.grid(True, ls=':') 

39 

40 plt.subplot (2, 2, 2) 

41 plt.plot(t, X, color= 'darkblue' , Iw = 2) 
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42 plt.xlabel( 'Time (t)', fontweight= 'bold' ) 

43 plt.ylabel( 'Preys in wood (x(t))', fontweight= 'bold' ) 

44 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

45 mnx = np.floor (100*min (x))/100 

46 mxx = np.ceil (100*max (x))/100 

47 plt.yticks(np.arange(mnx, mxx+mxx/1000, (mxx-mnx)/10), ... 

fontweight= 'bold' ) 

48 plt.grid(True, ls=':') 

49 

50 plt.subplot (2, 2, 3) 

51 plt.plot(t, y, color= 'crimson' , Iw = 2) 

52 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

53 plt.ylabel (' Spiders (y(t))', fontweight= 'bold' ) 

54 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

55 mny = np.floor (100*min (y))/100 

56 mxy = np.ceil (100*max (y))/100 

57 plt.yticks(np.arange(mny, mxy+mxy/1000, (mxy-mny)/10), ... 

fontweight= 'bold' ) 

58 plt.grid(True, ls=':') 

59 

60 plt. subplot (2, 2, 4) 

61 plt.plot(t, z, color= 'green' , Iw = 2) 

62 plt.xlabel (' Time (t)', fontweight= 'bold' ) 

63 plt.ylabel (' Preys in vineyard (z(t))', fontweight= 'bold' ) 

64 plt.xticks(np.arange(0, T+T/10, T/10), fontweight= 'bold' ) 

65 mnz = np.floor( 100*min (z))/100 

66 mxz = np.ceil (100*max (z))/100 

67 plt.yticks(np.arange(mnz, mxz+mxz/1000, (mxz-mnz)/10), ... 

fontweight= 'bold' ) 

68 plt.grid(True, ls=':') 

The solution of the pest optimal control problem with Gekko is explained 
in Figiire 11.5. 
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